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ABSTRACT 



This project paper reports the interobserver 
agreements and reliabilities for the observation procedures used in 
the Descriptive Study of Phase IV of the Individually Guided 
Education Evaluation Project. Only data from four observers—at the 
two Developing Mathematical Processes Schools and the two Wisconsin 
Design for Reading Skills Development Schools, grades 2 and 5— were 
used interobserver agreements and reliabilities were computed tor 
each'of three situations: training session, retraining session, and 
on-site visits. For each pair of observers and for each Category, 
marginal agreements were computed. For the category sets— General 
Content, Pacing, Grouping, Materials, Learner Moves , and 
interactions-Category Marginal Agreements and Cohen s Kappa were 
computed. Generalizability coefficients for estimates of reliability 
were computed for the training and retraining sessions for groups of 
three observers observing two or three classes for two days. The 
results indicated that thjre was high interobserver agreement at the 
three different check times on the categories of self-paced, 
other-paced, individual, large group, paper/pencil, engaged-written, 
engaged-covert, and non-engaged off task. The low interobserver 
agreements were explained mainly as a result of a low requency o 
occurrence. The generalizability coefficients for total engagement 
and total non-engagement were acceptable. (Author/BSI 
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MISSION STATEMENT 



The mission of the Wisconsin Research and Development Center 
is to improve the quality of education by addressing the lull 
ranqe of issues and problems related to individualized schooling. 
Teachinq, learning, and the problems of individualization are 
qiven concurrent attention in the Center's efforts to discover 
processes and develop strateqies and materials for use in the 
schools. The (.'enter pursues its mission by 

• conducting and synthes l zi nq research to clarify tin- 
processes of school-aqe children's learning and 
development 

• conductinq and synthesizinq research to clarify effective- 
approaches to teaching students basic skills and concepts 

• developing and demonstrating improved instructional strategies, 
processes, and materials for students, teachers, and school 
administrators 

• providing assistance t«> educator:; w,hich help.-, transfer the 
outcomes of research and development to improved practice 
in local schools and teacher education institutions 

The Wisconsin Research and Development Center is supported 
with funds from the National Institute of Education and the 
University of Wisconsin. 
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Abstract 



This project paper reports the interobserver agreements and 
reliabilities for the observation procedures used in the Descriptive 
Study of Phase IV of the IGE Evaluation Project. Only data from 
observers at the two DMP and two WDRSD schools are reported in this 
paper. The four observers attended a four day training session, 
January 10-13 1978, to learn how to use the observation procedures. 
On March 16 and 17 a retraining session was held at the middle of the 
fourteen week investigative period to check on the interobserver 
agreements. In addition, on-site visits were made to each school by 
one of the Phase IV staff where both the observer and staff member did 
one or two observations together. Interobserver agreement and reliabil- 
ities were computed for each of these three situations — training 
session, retraining session, and on-site visits. For -each pair of 
observers and for each category, marginal agreements were computed. For 
the sets of categories — General Content, Pacing, Grouping, Materials, 
Learner Moves and Interactions — Category Marginal Agreements and 
Cohen f s Kappa were computed. Generalizability coefficients for estimates 
of reliability were computed for the training and retraining sessions 
for groups of three observers observing two or three classes for two 
days. The results indicated that there was high interobserver agreement 
at the three different check times on the categories of self -paced, other- 
paced, individual, large group, pdfrer/pencil, engaged-written, engaged- 
covert, and non-engaged off task. The low interobserver agreements 
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were explained mainly as a result of a low frequency of occurrence. 
The generalizability coefficients for total engagement and total 
non-engagement were acceptable. 
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Introduction 



As a comprehensive system of education, Individually Guided 
Education (IGE) has as a goal the development of self -direction and 
motivation for learning in students, as well as the instruction of 
students at different levels of achievement. To examine IGE as an 
operating system, a five-phase plan was developed (Romberg, 1976). 
In Phase I, the extent to which key features of IGE had been implemented 
was assessed for a large sample of schools and related to both pupil 
and staff outcomes. In Phase II, more complete implementation data 
were collected from a sample of thirty schools to both verify and 
extend the Phase I data. Phase III was designed to examine the social 
meanings of schooling as interpreted within IGE schools on a day-to-day 
basis. Phase IV investigated the use of the Center's three curriculum 
programs in IGE and non-IGE situations. Finally, Phase V will provide 
a discussion.of the implications of the overall project to significant 
contemporary issues about the conception and study of schooling. 

Phase IV was a two-part study: the Descriptive Study and the 
Comparative Study. The purpose of the Descriptive Study was to describe 
the use of the Wisconsin Design f or Readin g Skills Development (WDRSD) , 
(Otto, 1977); Developing Mathematical Processes (DMP), (Romberg, 1977); 
and Prereading Skills (PRS), (Venezky & Pittelman, 1977). Two schools 
using WDRSD, two schools using DMP, and three schools using PRS partici- 
pated in the Descriptive Study from January to May, 1978. The purpose 
of the Comparative Study was to compare the use of WDRSD and DMP in IGE 
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and non-IGE situations and to contrast these programs with alternative 
programs being used in IGIi istuations. Math classes at eight schools 
and reading classes at eleven schools participated in the study from 
October 1978 until May 1979. A more detailed description of the design 
of Phase IV and the curriculum programs is given in Webb and Romberg 
(1979). 

As in the other phases of the IGE Evaluation Project, only students 
at grade levels 2 and 5, or the IGE equivalent, participated in the 
math and reading skills portion of the study. The two schools that 
were selected to be in the reading skills portion of the Descriptive 
Study were both IGE schools using WDRSD at these grade levels. Both 
schools initially implemented the WDRSD at these grade levels. Both 
schools initially implemented the WDRSD program in 1971. While the 
Word A*ttack, Study Skills, and Comprehension elements were taught at 
both grade levels at School 452, only Word Attack was used at Grade 2, 

and Study Skills Was used at Grade 5 at School 504. 

The two DMP schools that were selected to be in the Descriptive 
Study' were both IGE schools using DMP at these grade levels. Each 
school had been using DMP at the primary level for at least two years. 
The grade 5 DMP materials had only been available for approximately 
a year prior to the study. These schools then had only been using DMP 
at grade level 5 for one year. 

The three main variables investigated in Phase IV were pupil 
outcomes, opportunity to learn, and means of instruction. Pupil out- 
comes were measured in two ways, uiiing achievement monitoring and domain 
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referencing tests. The development of these tests is described in 
Webb (,1979a) and Dunhant, Nerenz, and Webb (1979). Information on the 
other two variables was collected using observations (Webb & Nerenz, 1979a) 
and teacher logs (Webb & Nerenz, 1979b). 

;he data collection- pha^e of the Descriptive Study lasted seventeen 
weeks, beginning' January 16 and ending May 19, 197S. This time was 

divided into two periods of seven weeks, period one, January 23 to 

■4, 

March 10, and period two, March 27 to May 12. Tests were administered 
during the weeks of January 16, March 13, and May 15. This project 
paper discusses the interobserver agreements of the two DMP and two 
WDRSD observers in using the observation system for the training and 
retraining sessions and the on-site visits. 
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Observations 



V 



In order to obtain information about the means of instruction 
and use of time during instruction in mathematics and reading skills 
in Grade 2 and Grade 5 classrooms or units using the DMP and WDRSD 
curriculum programs, observations were conducted approximately every 
week during the seventeen-week investigative period. The procedures 
used in conducting these observations are reported in detail in Project 
Paper 79-32 (Webb, 1979 b) and summarized briefly below. 

, Within each classroom or unit, six target students. who were 
either randomly selected from the entire class or from a stratified sample 
within the units were observed using a time sampling procedure. The 
activity that a target student was participating in was observed for an 
instant and recorded as an event on the observation form, requiring 
approximately thirty seconds. This was followed by another thirty- 
second period for the observation and recording of the next student's 
activity. Each target student was thus observed in sequence. Then 
general comments were recorded concerning the whole class and requiring 
another thirty seconds. This cycle was repeated from the designated^ 
beginning of the reading skills or mathematics period until the end of 
that instructional period. The set of categories used to record informa- 
tion included: 
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Time not spent on the content area of the 
subject matter of the instruction. 
Indication of the objective or skill of 
the student's activity. 

Whether the student or another individual 
controls the pace of instruction. 
The size of the group of which the student 
is a part. 

The materials used in the activity. 
The student's engagement or non-engagement 
in the activity. 

Verbal interactions involving the student 
during the activity. 
The total amount of time spent by the six target students in each 

of these categories during the observation period was summed, providing 

r 

an estimate of the allocated and available time. These observation cat- 
egories, procedures, and means of analysis are similar to those used 
in the Beginning Teacher Evaluation Study (Marliave, et al., 1977). 



General Content - 

Specific Content - 

Pacing - 

Grouping - 

Materials - 
Learner Moves - 

Interactions - 



II 



Observers and Observer Training 



All of the observers were from the area in which the school was 
located and had had some previous elementary teaching experience. Most 
had served as teachers, substitute teachers, or aides in the school 



15 



where the observations were to be conducted. A more detailed descrip- 
tion of each observer is given in, Project Paper 79-42 (Webb & Romberg, 1979) 

A description of the training and re-traininfe sessions is given 
in Project ^Paper 79-33 (Webb & Nerenz, 1979c). Briefly, the DMP and WDRSD 
training session 'lasted for four days. The first day .was spent providing 
the observers with an introduction to the system and a review of the 
curriculum ^terials. The next three days were spent doing observations 
in groups of three, one expert and two observers, in actual classrooms. 
On each of these days,, three classes were observed — one fifth-grade DMP 
class, one second-grade DMP class, and one secopd-grade WDRSD class. 
During these sessions, the two observers at the DMP schools ^were always 
grouped together, as were the two observers at the WDRSD schopls. The 
expert observers did observations with both pairs of observers at least 
once so that each pair had some experience with each expert. The training 
session was designed to familiarize the observers with the system, provide 
them with observation experiences, and measure interobserver agreement. 

Whenever two or more observers observed together, certain procedures 
were followed. One clock was designated as the indicator of the time 
intervals. If a wall clock with a second hand was not visible to all of 
the observers, one of the observers acted as time keeper and provided a 
signal when the next target student was to be observed. A period of thirty 
seconds was allowed to observe each target student within one cycle, with 
each cycle lasting 3-1/2 minutes. Communication between the observers 
was kept to a minimum and was used mainly to designate the beginning of 
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another cycle. Observations were conducted for the tj>tal class period, 
after which the observers reviewed the coding of eacn\event an^ checked 
on their agreement. Discrepancies were discussed at thl^^ime 
From the training session, several sources of error were 
considered. The definition of a category may be ambiguous, making 
it difficult to distinguish among behaviors that fit the category. 
In addition, student activities may be ambiguous, allowing for 
more than one interpretation of the activity. How the student is en- 
gaged can frequently be interpreted in more than one way, with very 
little difference in certain situations between non-engagement in an 
interim activity and covert engaged behavior. Such difficulty in 
classifying student activities is also increased due to the fact that 
student activities are constantly changing. Thus, variations in the 
instant during whicji.the two observers view the target child represent 
another source of error which will influence the level of inter-obser- 
ver agreement. .This problem results in observer error when observa- 
tions are compared event by event, but should be minimized when class 
marginal totals across students and events are tabulated. 

The training of the observers was designed to reduce as many 
of these errors as possible. The categories were defined with examples 
in the observer's manual. Ambiguous situations were discussed and 
• clarified. By the end of the training session, it was expected that 
each observer would have marginal agreement with an expert on each 
category of .90 or above. The generalizability coefficients were 
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used to indicate the degree of consistency among the raters without 
reference to particular level of agreement. • * 

One problem in conducting observations over a. period of time is 
the vigilance of the observers in performing the observations at the 
same level of competency at the end as at the beginning of the in- 
vestigative period. To minimize this problem, all of the DMP and WDRSD 
observers attended a retraining session between the two halves of Jie 
investigative period. This two-day session was spent doing observations 
in groups of three, similar to the training session, to check on inter- 
observer agreement. This same procedure of conducting observations 
with two or more observers was followed for this retaining session, and 
observer pairs and experts observed the same classes at the same schools 
for which training data were collected. In addition, on-site visits 
were made by Phase IV staff members to each school at least once during 
each period. This was a check of the observer's application of the 
coding system within the actual setting where data was being collected. 
The visit also was used to determine whether the observers were main- 
taining or improving the level of intercoder agreement which was ' 
reached at the training session. 

All of the observations during the training $nd retraining ses- 
siona for DMP and WDRSD observers were conducted in actual classrooms 
under conditions which would be very similar to those which each observer 
would- experience in obtaining the data. Actual classrooms were used in 
the training for several reasons. First, the observation system and 
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and procedures were complex enough that the activity of the target 
student could only be interpreted and coded in terms of the previous 
activities and behaviors, as well as the relationship with the teacher 
and the other students. To use • J *o tapes in the training sessions 
would require a high quality of video taping using a split scree i 
focusing on both the target student and the larger classroom setting, 
including the teacher and other students. This would have required 
substantially more preparatory time and equipment than was available. 

A second reason for using the actual classroom was that the 
ratio of trainers to trainees was small, 1 to 2. This provided an 
adequate amount of time for the trainer to discuss in detail with the 
observers any discrepancies in the coding of classroom events following 
each observation. The discussions included information about the 
teacher, the target student, other students, and the. materials, all 
of which were to be taken into account in coding the activities of the 
target children. 

One disadvantage of conducting the training in actual classrooms, 
however, is that the observers may not be exposed to all possible 
situations. In an attempt to minimize this restriction in the training 
experiences, for the observers for DMP and WDRSD, sessions were observed 
in three different types of classes: one Grade 5 DMP class, one Grade 2 
WDRSD class, and one Grade 5 WDRSD class. This exposed the observers 
to a number! of situations. In addition, after the observers returned to 
their schools, one of the trainers Visited each of them and did an 
on-site observation, discussing the coding of any specific situations 
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which had- not occurred during the training session but which the 
observer had subsequently experienced. 



training of observers is the difficulty in separating unwanted observer 
errors in the coding of unambiguous examples- from expected observer 
disagreements on ambiguous events. A detailed discussion followed each 
observation among the two observers and the trainer. JThis time was 
used to identify observer disagreements and to determine the reason for 

ft 

such conflicts. Disagreements due to ambiguous events were noted to be 
considered in revising the definition of the categories for the com- 
parative study. It was not necessary to have exact agreement on each 
event since the observation data was processed by summing across the 
events for the total class period* This lessened the need for agreement 
on each event as long as the marginal agreements were reasonable. 

Using actual classrooms during the training sessions, then, was 
selected as the most reasonable means of training the observers because 
of the small number of observers to be trained, the time available 
to prepare quality video-tapes, or instructional materials, and the 
ability to discuss each observation in detail after the observation. 
Also, varied types of sessions were observed and observations were 
analyzed to provide information about the class, meaning that fre- 
quencies were computed across events and students and allowing for 
some disagreement between observers on specific events. 



Another difficulty in using actual classroom settings for the 
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Interobserver Agreement and Reliability of Observations 

In obtaining information about the means of instruction and use 
of time, the observers must have a thorough knowledge of the observation 
system in order to interpret students' classroom activities and record 
these activities using the appropriate categories. Secondly, in order 
to have reliable data, the observers must be able to use the observation 
system consistently across observation days. Thus, measures of inter- 
observer agreement and consistency over time were needed. 

Knowledge of the Observation System # 

The assessment of the observers' knowledge of the categories and 
procedures in. the observation system was obtained by comparing the 
results of their observations by category for actual classroom sessions 
with those of an expert observer— one of the two Phase IV staff members 
who developed the observation system. The coding by the expert observer 
was used as the criteria for the observers to match. The level of 
agreement among the two expert observers was checked prior to the observer 
training session. This procedure of measuring the observers' knowledge 
of the system worked adequately for categories which were frequently 
observed such a3 pacing categories, individual or large groups, paper 
and pencil materials, and learner moves. For the categories that occurred 
infrequently, the knowledge of the observers was not tested during the 
training session When they occurred. These categories were given more 
emphasis during the on-site visits • 

21 
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Reliabilites across Situations 

Each DMP and WDRSD observer was to do observations at both 
Grades 2 and 5 from twelve to sixteen times over fourteen of the 
seventeen weeks composing the investigative period. In each case, 
the. observer usually observed the same group of students using the same 
curriculum materials at approximately the same time of day. The only 
exception was at one of the WDRSD schools where students were regrouped 
approximately every two weeks .and thus the target students were not 
always in the same instructional group. At this school, the observations 
were made on the instructional group containing the largest number of 
target students, with additional children observed so that the total 
number of students that were observed during each session was six. 

Each observer then did observations on different days and at both 
grades two and five. In order to get an estimate of the reliability of 
observers across situations a generalizability coefficient (Cronbach, 
Gleser, Nanda, and Rajaratnam, 1972) was computed. Variance due to day 
was treated as a fixed effect in that some variance on each category 
was expected within a class from day. to day. The variance components 
were computed using BMD: Biomedical Computer Programs (Dixon, 1971). 
For computing the generalizability coefficients for the WDRSD and DMP 
training session, data were used from the observations of two groups of 
two observers and one expert for three classes, one on Thursday and two 
on Friday. This provided, for each category, an estimate of generaliz- 
ability for each set of observers and indicates the consistency among 
the observers across different classes. 
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IV 



Analysis and Results 

Interobserver Agreement 

Marginal agreement (Frick and Se&jnel, 1978) was used as the single 
measure of interobserver agreement in comparing each observer with the 
expert for specific aspects of each observation category. This agreement 
is a simple percentage relating the total frequency of a category across 
all the events for all of the target students for the two individuals. 
This measure of agreement was chosen since the unit of analysis is the 
class or I & R unit and the major analyses will use marginal totals. 
Marginal agreement was calculated for each category that had a sufficient 
number of observed events to make the measure meaningful. Calculations 
were generally not done for the general content categories, some of the 
materials categories, and some of the interactions because of low 
frequencies during the observed sessions. 

Two measures of interobserver agreement were used in determining 
the accuracy of the observers compared with the expert for the set 
of categories, such as pacing, grouping, and learner moves. The 
overall marginal agreement is the mean of the individual marginal 
category agreements for all of the sub-categories in the set. 

In addition, Cohen's kappa (k) was computed. This coefficient 
takes into consideration chance agreement and is based on observed 
marginal distributions (Frick and Semmel, 1978). Cohens k 
coefficient was selected from among the other coefficients since 
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it does not require that the marginals be symmetrical and propor- 
tional to known populational marginals. Cohen's k can also be 
tested for the significance of the difference between two indepen- 
dent K's and for the null nypothesis. However, the knowledge that 
K is beyond chance is of little value, since, as an agreement co- 
efficient, the desired value is much greater than 0. The value 
of k indicates the proportion of events observed that are in agree- 
ment, with chance agreement excluded. One assumption for the use 
of k is that the number of events is large— greater than or equal 

t0 100 in order to assure that the proportion of events for which 

agreement is expected by chance remains nearly constant. Cohen's k 
and the overall marginal agreement were used to determine the accur- 
acy of the observers during the training session, the retraining 

« 

session, and the on-site visits. 

Two other measures of interobserver agreement were calculated 

and are included on the tables in the appendices but are not used 
in the analysis. These measures were calculated to be compared with 
the marginal agreement and to judge their appropriateness in measur- 
ing the agreement between observers for t{ie data collected. One 
measure is the nominal agreement which compares two observers on 
individual events. This is a stronger criterion than marginal 
agreement and was not used in the analyses since only marginal 
scores were used from the observations. Nominal agreement does give 
an indication of the agreement between two observers in recording 
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an instant of an activity rather than describing the total events 
over the activity. 

The second measure included is Light's extension of k (Frick 
and Semmel, 1978). This coefficient is algebraically equivalent to 
Cohen's K, but is more versatile than the latter. Using the Light's 
extension of K it is possible to provide for one category a measure 
of the conditional agreement with two observers which compares the 
agreement of one observer's score with a criterion score — the 
expert's score. The consideration of chance agreement is included 
in this coefficient. This coefficient was calculated to explore 
what additional information it can provide beyond the marginal agree- 
ment and Cohen's K for the set of categories. 

Generalizability of Observation s 

The data from the training and retraining sessions for the DMP 
and WDRSD observers were used to perform small generalizability 
studies and to calculate generalizability coefficients (p*) on some 
of the observation categories— pace, group, learner moves, and 
paper and pencil materials. During the training session two groups 
of three observers each observed three classes. The generalizability 
coefficients calculated from this data'provide some measure of the 
ability of the observers to discriminate among classes. A high 
coefficient indicates that a large amount of the variance can be 
accounted for by differences among classes and that the variance 
due to the differences among observers is relatively small. A low 
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coefficient is due to large differences among the raters relative to 
the variance among classes. A zero coefficient indicates there is 
no variance among the classes. 

In the retraining session for the DMP and WDRSD observers a cross 
design was used for two sets of observers. One set was composed of 
DMP observer 40 and expert observer 80. The other set consisted 
of the two WDRSD observers, 10 and 20, and expert observer 90. 
DMP observer 30 was only able to attend the last day of the retrain- 
ing. Each set of observers observed the same three classes on two 
consecutive days resulting for the first set a 2 x 3 x 2 (observer 
x class x day) design and for the second set a 3 x 3 x 2 (observer x 

class x day) design. The BMP; Biomedical Computer Program 08V 
(Dixon, 1971) was used in calculating the variance components. Day 
was used as a fixed effect. The generalizability coefficient was 
computed using the equation: 

a 

when 4 q - variance component for class 

q' = variance component for the interaction between 

teacher and class 

Generalizability coefficients for the retraining session were 

' ril I'M I In- fi.mii- iili'in V.ll Inn i ;ili- K f,| ;IM li.J I lie (Kilning 

session. A large coefficient .tor the retraining session denotes 
dial the rl.isj; w.ts t lu« prliiiarv iUMinv ol variance relative to the 
variance due to interactions among the class and the two variables 
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rater and day. A low or zero coefficient can be interpreted similar 
to the interpretation of the coefficients for the training session. 
The coefficient should- be higher for the retraining session since 
the observers are more experienced. 
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Interpretations 



Training Session 

Observer Agreement . The summary of the observer agreement 
coefficients by sets of categories for the DMP and WDRSD training 
sessions are given in Table 1 through Table 9. Listed in the tables 
for each observer -expert pair and for each category is the marginal 
agreement. The overall marginal agreement and Cohen's kappa (k) are given 
for the category set for each observer-expert pair. The contingency 
tables showing the frequencies for each category in each set for the 
training session are included in Appendix A. These tables can be 
consulted in order to determine the extent that the categor> observed 
and the actual frequency of disagreement between the expert and observer. 

The agreement coefficients for the pace categories are given in 
Table 1. The range of marginal agreements for self -paced is generally 
from .86 to 1.00. During one Grade 2 class using WDRSD, the expert 
coded six instances of self-pace which neither of the two observers 
noted. This resulted in a marginal agreement of zero, but is of little 
significance because .of the low frequency of occurrence in this case and 
the consistently high agreements on the other occasions. The marginal 
agreements for other-paced generally ranges from .68 'to 1.00. Two 
occasions with low agreement, observer pairs 30-80 and 40-90 on Friday, 
can be accounted for by a relatively low number of positive occurrences 
for other-paced coded during the class. The marginal agreements for not 
coding either of the pace categories vary greatly with low agreements 
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TABLE 1 



Summary of the Category Marginal, Overall Marginal 
and Cohen's Kappa Observer Agreement Coefficients for 
the DMP and WDRSD Observers with the Expert Observers 

r 

Training Session: January 12 and 13, 1978 



Pace Categories 



Observer 
Pair 

Obs a Expert 



Curriculum Total 
and Eventa 
Grade Observed 



Category Marginal 



Self-Paced 



Other-Paced 



No Pace 



Thursday, January 12, 1978 



Overall Cohen 
Marginal Kappa 
(K) 



10 


90 


WDRSD 


2 


54 


.86 


(30) b 


.79 


(24) 


- c (o) 


.82 


.81 


20 


90 


WDRSD 


2 


54 


.94 


(30) 


.79 


(24) 


- (0) 


.86 


.64 


30 


80 


WDRSD 


2 


60 


1.00 


(34) 


.83 


(15) 


.73 (1) 


.85 


.86 


40 


80 


WDRSD 


2 


60 


1.00 


(34) 


.88 


(15) 


.82 (1) 


.90 


.88 


10 


80 


DMP 


2 


84 


.94 


(34) 


.94 


(31) 


1.00(19) 


.96 


.76 


20 


80 


DMP 


2 


84 


.90 


(34) 


.87 


(31) 


1.00 (19) 


.92 


.78 


30 


90 


DMP 


2 


78 


.86 


(48) 


.68 


(22) 


.88 (8) 


.80 


.61 


40 


90 


DMP 


2 


78 


.78 


(48) 


1.00 


(22) 


.88 (8) 


.95 


.78 



Mean 


.94 


.85 


.88 


.88 


.76 






Friday, 


January 13, 1>78 






• 





10 


90 


DMP 


5 


30 


.93 


(27) 


- (0) 


.60 


(3) 


.76 


.71 


20 


90 


DMP 


5 


30 


.96 


(27) 


- (0) 


.67 


(3) 


.82 


.78 


30 


80 


DMP 


5 


' 30 


1.00 


(ID 


1.00 (12) 


1.00 


(7) 


1.00 


1.00 


40 


80 


DMP 


5 


30 


1.00 


(11) 


.92 (12) 


.86 


(7) 


.93 


.95 


10 


90 


WDRSD 


2 


54 


.00 


(6) 


.92 (47) 


.33 


(1) 


.42 


.27 


20 


90 


WDRSD 


2 


54 


.00 


(6) 


.92 (47) 


.33 


(1) 


.42 


.06 


30 


80 


WDRSD 


2 


54 


.89 


(47) 


.40 (2) 


.71 


(5) 


.67 


.40 


40 


80 


WDRSD 


2 


54 


1.00 


(47) 


1.00 (2) 


1.00 


(5) 


1.00 


1.00 


10 


80 


DMP 


2 


78 


.97 


(32) 


.95 (41) 


.62 


(5) 


.85 


.82 


20 


80 


DMP 


2 


78 


1.00 


(32) 


.98 (41) 


.80 


(5) 


.92 


.88 


30 


90 


DMP 


2 


84 


.98 


(65) 


.82 (11) 


.73 


(8) 


.84 


.41 


40 


90 


DMP 


2 


84 


.92 


(65) 


.36 (ID 


.89 


(8) 


.72 


.60 




Mean 








.80 




.83 


.71 




.78 


.66 



a Observers 10 and 20 observed at WDRSD schools. 
Observers 30 and 40 observed at DMP schools. 

b Events coded by the expert. 

c - No events were coded for this category by either 
the observer or expert. 
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explained by low frequency of occurrences. 

The overall marginal agreement for pacing, the average of the 
three category .urginals, ranges fro. .42 to 1.00. The low overall 
marginals correspond to occasions where one of the categories had a 
low frequency. Except for four of the occasions, the Cohen's K 
coefficients range from .60 to 1.00. The four occasions with low k's 
all occurred on Friday and three while observing a Grade 2 class using 
WDKSD . Again part of the reason for these coefficients is the dis- 

.,„„„„ which was observed infrequently 
agreement between observers on a category whlcn was 

by the expert. Overall, the observers had high agreement on the pacing 
categories. One observer, 30, had a few more problems with the coding 
of pacing than did the other. Also, there appears to be - difficulty 
with one class of Grade 2 WDRSD. The class differences will be studied 
i„ more detail in the generalizability study for the retraining sessions. 

The coefficients for the grouping categories are given in Table 2. 
Most of the student'available time was coded as being spent in individual 
„„rk or in large groups. These two categories correspond closely to 
self -paced and other-paced, respectively. The marginal agreements for . 
individual grouping are generally high ranging from .64 to 1.00. The 
range for large groups is also high, ranging in general from .67 to 1.00. 
Those coefficients below this range occurred where a very low frequency, 
of the category was observed. The agreements in observing pairs and 
small groups are less stable than for the other two groupings. One reason 
for this is the low frequency of occurrence of these groupings. However, 
even considering this fact it appears that the observers had more difficulty 
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Summary of the Category Marginal, Overall Marginal 
and Cohen's Kappa Observer Agreement Coefficients for 
the DMP and WDKSD Observers with the Expert Observers 

Training Session: January 12 and 13, 1978 
Grouping Categories 



Observer 
Pair 

Obs a Expert 



Curriculum 
and 
Grade 



Total 
Events 
Observed 



C ategory Marginal 



Ind. 



Pair 



SraGR 



LgGR 



Other 



Overall Cohen's 
Marginal Kappa 
(K) " 



Thursday, January 12, 1978 



10 


90 


WDRSD 


2 


54 


.64(20) b 


.40 


(5) 


.62(13) 


.81(16) 


- c (0) 


.62 


.64 


20 


90 


WDRSD 


2 


54 


.74(20) 


.40 


(5) 


.77(13) 


.75(16) 


- (0) 


.66 


.68 


30 


80 


WDRSD 


2 


60 


.94(31) 


60 


(3) 


- (0) 


.83(15) 


.73 (11) 


.77 


.82 


AO 


80 


WDRSD 


2 


60 


.90(31) 


.50 


(3) 


- (0) 


.88(15) 


.82 (11) 


.78 


.82 


10 


80 


DMP 


2 


84 


.96(26) 


.67 


(9) 


1.00(11) 


.83(19) 


1.00 (19) 


.89 


.80 


20 


80 


DMP 


2 


84 


.92(26) 


.78 


(9) 


.79(11) 


.95(19) 


1.00 (19) 


.89 


.83 


30 


90 


DMP 


2 


78 


.74(38) 


.75 


(8) 


.43(14) 


.80(10) 


.88 (8) 


.72 


.58 


40 


90 


DMP 


2 


78 


.86(38) 


1.00 


(8) 


.57(14) 


.91(10) 


.88 (8) 


.84 


.77 



Mean .84 .64 .70 .84 ;98 .77 .74 



Friday, January 13, 1978 



10 


90 


DMP 


5 


30 


1.00(24) 


1.00 (1) 


.00 


(1) 


.00 (1) 


.60 (3) 


.52 


.61 


20 


90 


DMP 


5 


30 


.86(24) 


.00 (1) 


.00 


(1) 


.00 (1) 


.67 (3) 


.30 


.46 


30 


80 


DMP 




30 


.82 (9) 


.00 (1) 


.00 


(1) 


1.00(12) 


1.00 (7) 


.56 


.90 


40 


80 


DMP 


5v- 


30 


.90 (9) 


.00 (1) 


1.00 


(1) 


.92(12) 


.86 (7) 


.74 


.90 


10 


90 


WDRSD 


2 


54 


.00 (1) 


- (0) 




(0) 


.98(52) 


.33 (1) 


.44 


.59 


20 


90 


WDRSD 


2 


54 


1.00 (1) 


- (0) 




(0) 


.96(52) 


.33 (1) 


.76 


.14 


30 


80 


WDRSD 


2 


54 


.95(41) 


.50 (1) 




(0) 


.50 (2) 


.71 (5) 


.67 


.51 


40 


80 


WDRSD 


2 


54 


1.00(41) 


1.00 (6) 




(0) 


1.00 (2) 


1.00 (5) 


1.00 


.91 


10 


80 


DMP 


2 


78 


.97(31) 


1.00 (1) 


1.00 


(4) 


.95(37) 


.62 (5) 


.91 


.84 


20 


80 


DMP 


2 


78 


1.00(31) 


1.00 (1) 


1.00 


(4) 


.97(37) 


.80 (5) 


.95 


.85 


30 


90 


DMP 


2 


84 


.65(41) 


.43(23) 


0.00 


(9) 


G.00 (3) 


.72 (8) 


.36 


.41 


40 


90 


DMP 


2 


84 


.66(41) 


.39(23) 


.22 


(9) 


.67 (3) 


.89 (8) 


.57 


.50 ., 



Mean .82 .53 .40 .66 .71' .65 .64 



a Observers 10 and 20 observed at WDRSD schools. 
Observers 30 and 40 observed at DMP schools. * 

b Events coded by the expert. 

c - No events were coded for this category by either 
the observer or expert. 
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in distinguishing when students were in pairs or small groups. Con- 
sidering only the occasions where the positive instances of the grouping 
observed by the expert was at least 10% of the total number of events, 
the range of marginal agreement for pairs is .39 to 1.00 and for small 
groups is 0.00 to 1.00. The lowest agreements occurred on Friday with 
the observer pairs 30-90 and 40-90 for a Grade 2 DMP class. The expert 
coded a higher frequency of pairs and small groups in this class than in ~ 
any of the others. However, the observers did not code these two 
categories with any larger frequency than in other classes which indicates 
that the observers still lacked, experience in identifying the use of pairs 
and small groups. 

The overall marginals for the set of categories are influenced by 
disagreements where there are a low frequency of a category coded^ In 
these cases the Cohen's K appears to be more representative of the 
situation. The Cohen's K ranges from .58 to .83 on Thursday and from 
.14 to .91 on Friday. The K - .14 occurred when there was a very high 
agreement on one of the categories, large group, and a very low frequency 
. of occurrence for all of the other categories. This increased the 

factor due to chance agreement and lowered the value of K. Any difference 
in agreement on one of the 54 events has a large effect on the value of 
the kappa. In general the k's for the set of grouping categories is 
reasonably high which indicates that even though the observers did not 
achieve great proficiency in coding pairs and small groups, overall 
they demonstrated an ability to code grouping since mainly large groups 
and individual work was used. There do appear to be some differences 
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in agreement due to the expert. The observers consistently agreed more 
highly with expert 80 than with expert 90. Expert 80 developed the 
system and had the main responsibility for instructing the observers 
on the system, ■ The low agreement then appears to be partly due to 
the expert rather than just with the observers. 

Tables 3, 4, 5, 6 and 7 list the agreement coefficients for the 

material categories. Only p aper and p encil materials (Table 3) were 

coded as being used in all of the classes observed. For the other 
materials, the coefficients are only reported for the classes where their 
use was coded at least once. For paper and pencil materials the marginal 
coefficient is high (generally above ,90) 'as is the Cohen's K (generally 
above .60). Thus the observers demonstrated facility in coding the use 
of paper and pencil materials. Disagreements were due to the inter- 
pretation of when students were not actually using paper and pencil at 
the precise moment the event was coded but where paper and pencil were 
still part of the ongoing activity and should be coded. 

Manipulatives (Table 4) were observed, with one exception, only in 
DMP classes. There was higher agreement on Friday than on Thursday. 
The marginal agreements for DMP classes on Friday was generally above .80. 
The Cohen K f s for these ranged from .64 to .88 indicating the observers 
were adequate in identifying when manipulatives were being used. For 
the WDRSD Grade 2 class on Friday, Observer 20 coded playing bingo as 
using a manipulative rather than a game. This misinterpretation was 
disciiHttcd with the observer after the c la hh and clarified at that time. 

Games (Table 5) were observed being used very infrequently during 
the training session. For the Grade 2 WDRSD class on Friday, the marginal 
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Summary of the Category Marginal, Overall Marginal 
and Cohen's ."appa Observer Agreement Coefficients for 
the DMP and WDRSD Observers with the Expert Observers 

Training Session: January 12 and 13, 1978 
Paper and Pencil Material Category 



Observer 
Pair 

Obs a Expert 



Curriculum * Total 
and Events 
Grade ' Observed 



Category Marginal 



• 10 

20 
30 
AO 
10 
20 
30 
40 



10 

20 
30 
40 
10 
20 
30 
40 
10 
20 
30 
40 



90 

90 
80 
80 
80 
80 
90 
90 



Mean 



90 
90 
80 
80 
90 
90 
80 
80 
80 
80 
90 
90 



Paper /pencil 
used 



Paper /pencil 
v not used 



Other 



Overall 
Marginal 



WDRSD 2 
WDRSD 2 
WDRSD 2 
WDRSD 2 
DMP 2 
DMP 2 
DMP 2 
DMP 2 



DMP 5 
DMP 5 
DMP 5 
DMP 5 
WDRSD 2 
WDRSD 2 
WDRSD 2 
WDRSD 2 
DMP 2 
DMP 2 
DMP 2 
DMP 2 



54 
54 
60 
60 
84 
84 
78 
78 



30 
30 
30 
30 
54 
54 
54 
54 
78 
78 
84 
84 



Thursday, January 12, 1978 



.93 (39) 
.87 (39) 
.98 (49) 
.98 (49) 
.97 (61) 
1.00 (61) 
.90 (55) 
.90 (55) 



.80 (15) 
.40 (15) 
.00 (0) 



.00 
.67 
1.00 



(0) 
(4) 
(4) 



.67 (15) 
.67 (15) 



- c (0) 

- (0) 
.78 (11) 
.82 (11) 
1.00 (19) 
1.00 (19) 
.88 (8) 
.88 (8) 



.86 
.64 
.57 
.60 
.88 
1.00 
.81 
..81 



Cohen's 
Kappa 
(K) 



.85 
.40 
.60 
.72 
.78 
.89 
.57 
.81 



.94 


.53 


.89 




.77 


.70 


Friday, January 13, 1978 


.93 (27) 


. - (0) 


.60 


(3) 


.76 


.71 


1.00. (27) 


.00 (0) 


.67 


(3) 


.56 


.82 


.83 (23) 


.00 (0) 


1.00 


(7) 


.61 


.71 


.96 (23) 


.00 (0) 


.1.00 


(7) 


.65 


.82 


.00 (6) 


.92 (47) 


.33 


(1) 


.42 


.16 


.00 (6) 


.92 (47) 


.33 


(1) 


.42 


- .05 


.94 (47) 


.50 (2) 


.83 


(5) 


.76 


' .40 


.96 (47) 


.00 (2) 


1.00 


.(5) 


.65 


.82 


1.00 (45) 


.89 (28) 


.62 


(5) 


.84 


.81 


.98 (45) 


1.00 (28) 


.80 


(5) 


.93 


.78 


1.00 (73) 


.00 (3) 


.73 


(8) 


.58 


.85 


1.00 (73) 


.67 (3) 


.89 


(8) 


.85 


.95 



Mean 



.80 



.44 



.73 



.67 



.65 



a Observers 10 and 20 observed at WDRSD schools. 
Observers 30 and 40 observed at DMP schools. 

b Events coded by the expert. 

c - No Events were coded for this category by either 
the observer or expert. 
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TABLE 4 



Summary of the Category Marginal, Overall Marginal 
and Cohen's Kappa Observer Agreement Coefficients for 
the DMP and WDRSD Observers with the Expert Observers 

Training Session: January' 12 and 13, 1978 
* Manipulative Material Category 



Observer 
""pair 

Obs a Exp ert 



Curriculum 
and 
Grade ' 



Total 
Events 
Observed 



C ategory Marginal 



10 


90 


DMP 


5 


30 


20 


90 


DMP 


5 


30 


30 


80 


DMP 


5 


30 


AO 


80 


DMP 


5 


30 


10 


90 


WDRSD 


2 


54 


20 


90 


WDRSD 


2 


54 


30 


80 


WDRSD 


2 


54 


40 


80 


WDRSD 


2 


54 


10 


80 


DMP 


2 


78 


20 


80 


DMP 


2 


78 


30 


' 90 


DMP 


2 1 


84 


40 


90 


DMP 


2 


8.4 


Mean 



Manipulative 
Used 

Thursday, January 12, 1978 



Manipulative 
Not Used 



Other 



Overall , Coin's 
Marginal Kappa 
(K) 



10 


90 


WDRSD 


2 


54 




20 


90 


, WDRSD 


2 


54 




30 


80 


WDRSD 


2 


60 




40 , 


80 


WDRSD 


2 


60 




10 


80 


DMP 


2 


84 


1.00 


20 


80 


DMP 


2 


. 84 


.67 


30 


90 


DMP 


2 


78 


.77 


40 


90 


DMP 


2 


78 


.56 




Mean 








.75 



1.00 


(56) 


1.00 


(19) 


1.00 


.85 


.95 


(56) 


1.00 


(19) 


.87 


.87 


.87 


(47) 


.87 


(8) 


.84 


.50 


.64 


(47) 


.87 


(8) 


.69 


.51 


.86 




.94 




.85 


.68 



Friday, January 13, 1978 



.64 (14) 
.93 (14) 



.oa (0) 



.92 (23) 
.82 (23) 
.88 (49) 
.90. (49) 

.72 



.81 (13) 
.87 (13) 



.00 (53) 



.90 (50) 

.92 (50) 

.90 (27) 

.87 (27) 

.75 



.60 


(3) 


.68 


.67 


.67 


(3) 


.82 


.88 


.33 


(1) 


.11 


, .00 


.62 


(5) 


.82 


.64 


.80 


(5) 


.84 


.70 


.73 


(8) 


.83 


.73 


.89 


(8) 


.89 


.75 



.66 



.71 



.62 



■ • i" 

a Observers 10 and 20 observed at WDRSD schools. 
Observers 30 and 40 observed at DMP schools. 

b Events coded by the expert. 
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Summary of the Category Marginal, Overall Marginal 
and Cohen V. Kappa Obsorver Agreement Coefficients lor 
the DMF and WDkSD Observers with the Expert Observers 



raining Session: January 12 and 13, 1978 



ame 



Observer 
Pair 

.a 



Obs* Expert 



Curriculum 
and 
Grade 



Total 
Events 
Observed 



k 10 


90 


WDRSD 


2 


54 


20 


90 


WDRSD 


2 


54 


30 


80 


WDRSD 


2 


GO 


40 


80 


WDRSD 


2 


60 


10 


80 


DM? 


2 


84 


20 


80 


DMP 


2 


84 


30 


90 


DMP 


*2 


78 


40 


90 


DMP 


2 


78 




(]a t ££C) r y_ Marginal 



Game 
Used 



Game 
Not Used 



Thursd ay, January 12, 1978 



Other 



Overall Cohen's 
Marginal Kappa 
«) 



Mean 



10 


90 


DMP 


5 . ' 


30 


20 


90 


DMP 




30 


30 


80 


DMP 


h 


30 


40 


80 


DMP 




30 


10 


90 


WDRSD 


2 


54 


20 


90 


WDRSD 


2 


54 


30 


80 


WDRSD 


2 


54 


40 


80 


WDRSD 


2 


54 


10 


80 


DMP 


2 


78 


20 


80 


DMP 


2 


78 


30 


90 


DMP 


2 


84 


40 


90 


DMP 


2 


84 


Mean 



Friday, January 13, 1978 



.96 (53? 
.00 (53) 



.00 (3) 
.00 (3) 



.24 



-°(0) 
.00 (0) 



1.00 (73) 
.97 (73) 



.66 



.33 (1) 
.33 (1) 



.65 
.11 



.49 
.00 



.73 (8) 
.89 (8) 



.57 



.58 
.62 

.49 



.85 
.83 

.54 



ERIC 



a Observers 10 and 20 observed at WDRSD schools. 
Observers 30 and 40 observed at DMP schools. 

b Events coded by the expert. 

c -No Events were coded for this- category by either 
the observer or expert. 
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TABLE 6 

28 

Summary of the Category. Marginal, Overall Marginal 
and Cohen's Kappa Observer Agreement Coefficients for 
the DMP and WDRSD Observers with the Expert Observers 

Training Session: January 12 and 13, 1978 
Audio-Visual Aid 



Observer Curriculum Total 
Pa 

Obs a Expert 



Cate gory Margi nal Overall Cohen's 



„~ m other Marglnal 

Grade " Observed Used Not Used ________ 



10 


90 


WDRSD 


2 


5A 


30 


90 


WDRSD 


2 


5A 


80 


WDRSD 


2 


60 


AO 


80 


WDRSD 


2 


60 


10 


80 


DMP 


2 


8A 


20 


80 


DMP 


2 


8A 


30 


90 


DMP 


2 


78 


AO 


90 


DMP 


2 


78 



Mean 



10 


90 


DMP 5 


'30 


20 


90 


DMP 5 


30 


30 


80 


DMP 5 


30 


AO 


80 


DMP 5 


30 


10 


90 


WDRSD 2 


5A 


20 


90 


WDRSD '2 


5A 


30 


80 


WDRSD' 2 


54 


AO 


80 


WDRSD 2 


5A 


10 


80 


DMP 2 


78 


20 


80 


DMP 2 


,78 


30 


90 


DMP 2 


8A 


AO 


90 


DMP 2 


8A 


Mean 



Thursday, January 12, 1978 



Friday, January 13, 1978 



.00 (1) .97 (75) .73 (8) .57 .78 

.00 (1) 1.00 (75) .89 (8) .63 .88 

.00 .98 ,81 .60 .83 



* 37 

ERIC 



TABLE 7 



29 



Summary of the Category Marginal, Overall Marginal 
and Cohen's Kappa Observer Agreement Coefficients for 
the DMP and WDRSD Observers with the Expert Observers 

Training Session: January 12 and 13, 1978 
Printed Materials 



Observer 
Pair 

Obs a Ex pert 



Curriculum Total 
and Events 
Grade Observed 



10 
20 
30 
40 
10 
20 
30 
40 



90 
90 
80 
80 
80 
80 
90 
90 



Mean 



Mean 



2 
2 



WDRSD 
WDRSD 
WDRSD 2 
WDRSD 2 

DMP 

DMP 

DMP 

DMP 



54 
54 
60 
60 
84 
84 
78 
78 



Category M ar ginal 
Printed Mat. Printed Mat. 



Used 



Not Used 



Other 



Thursday, January 12, 19/8 

792 (25) b .94 (29) 

.88 (25) 1.00 (29) 

.00 (0) .98 (49) 

.00 (0) .80 (49) 

.00 (0) .99 (65) 



.36 



.94 



Friday, January 13, 1978 



10 


90 


DMP 


5 , 


30 


.00 (0) 


20 


90 


DMP 


5 


30 


.00 (0) 


30 


80 


DMP 


5 


30 


.00 (0) 


40 


80 


DMP 


5 


30 


.00 (0) 


10 


90 


WDRSD 


2 


54 




20 


90 


WDRSD 


2 


54 


.00 (0) 


30 


80 


WDRSD 


2 


54 


.85 (20) 


40 


80 


WDRSD 


2 


54 


.95 (20) 


10 


80 


DMP 


2 


78 




20 


80 


DMP 


2 


78 




30 


90 


DMP 


2 


84 




40 


• 90 


DMP 


2 


84 





.26 



.75 



.85 



Overall 
Marginal 



.76 



.73 



.59 



Cohen's 
Kappa 
(K) 



- C (0) 


.93 


.78 


- (0) 


.94 


.40 


,73 (11) 


.57 


.53 


.82 (11) 


.54 


.40 


1.00 (19) 


.66 


.83 



.59 



.82 (27) 


.60 


(3) 


.47 


.48 


1.00 (27) 


.67 


(3) 


.56 


.64 


.65 (23) 


1.00 


(7) 


.55 


.53 


.91 (23) 


.86 


(7) 


.59 


.68 v 


.00 (53) 


.33 


(1) 


.11 


.00 


.94 (20) 


.83 


(5) 


.87 


.67 


.97 (29) 


1.00 


(5) 


.97 


.97 



.57 



a Observers 10 and 20 observed at WDRSD schools. 
Observers 30 and 40 observed at DMP schools. 

b Events coded by the expert. 

e -No Events were coded for this category by either 
the observer or expert. 



ERIC 



38 



\ 



30 



TABLE 8 

Summary of the Category Marginal, Overall. Marginal 
and Cohen's Kappa Observer Agreement Coefficients for 
the DM1 1 and WURSD Ohservers with the Kxpert. Observers 

Training Session: January 12 and 13, 1978 
Learner Moves Categories 



Observer 
Pair 

Qbs a Expert 



Curriculum Total 
and Events 
Grade ' Observed 



EnW 



_C at eg o ry Ma rgijpal_ 
EnO EnC NI 



NW 



NO 



Other 



35 



Thursd ay T January 12, 1978 



10 
20 
30 
AO 
10 
20 
30 
40 



90 
90 
80 
80 
80 
80 
90 
90 



WDKSD 
WDRSD 
WDRSD 
WDRSD 
DMP 
DMP 
DMP 
DMP 



2 
2 
2 
2 
2 
2 
2 
2 



54 .54(11^ .29(2) .88(15) .86 (6) 

54 1 1.00(11) .33(2) .83(15) .83 (6) 

60 . 1.00 (9) .33(2) .81(21) .50 (2) 

60 .90 (9) .50(2) .91(21) .50 (2) 

84 .50(10) .38(3) .83(30) 1.00 (3) 

84 .59(10) .43(3) .50(30) .60 (3) 

78 .88(14) .38(3) .96(25) .75 (12) 

78 .82(14) .50(3) .86(25) 1.00 (12) 



.50(12) ,73(8) 
.08(12) .80(8) 
.86 (6) 1.00(9) 
.83 (6) .89(9) 
.67(12) 
.58(12) 
.14 (7) 
.29 (7) 



- c (0) .63 .27 

.00 (0) .64 .37 

.73(11) .75 .65 

.82(11) .76 .68 

.86(7)1.00(19) .75 .49 

.50(7)1.00(19) .60 .55 

.44(9) .88 (8) .63 .44 

.56(9) .88 (8) .70 .63 



Mean 


.78 .39 .82 


.76 .49 


.72 


.68 .51 




Friday, January 


13, 1978 




• 



10 


90 


DMP 


5 


30 


20 


90 


DMP 


5 


■30 


30 


80 


DMP 


5 


30 


40 


80 


DMP' 


5 


30 


10 


90 


WDRSD 


2 


54 


20 


90, 


WDRSD 


.2 


54 


30 


80 


WDRSD 


2 


54 


40 


80 


WDRSD 


2 


54 


10 


80 


DMP 


2 


78 


20 


80 


DMP 


2 


78 


30 


90 


DMP 


2 


84 


40 


90 


DMP 


2 


84 



.64(11) 
.65(11) 
.86 (7) 
30 1.00 (7) 
- (0) 
.00 (0) 
.80(12) 
.75(12) 
.62 (5) 
.80 (5) 
.47 (8) 
.53 (8) 



.00(0)1.00 (9) 
.00(0) .44 (9) 
- (0) .86 (6) 

* .00(0)1.00 (6) 
.00(0) .76(13) 
.00(0) .92(13) 
.50(4) .86(14) 

1.00(4)1.00(14) 
.50(3) .59(46) 
.43(3) .96(46) 
.71(5) .73(22) 
.71(5) .86(22) 



00(00) 1.00(22) 
* (01) .50(22) 
,50 (2) .86 (7) 
,50 (2) 1.00 (7) 
.88 (8) .80(30) 
.88 (8) .90(30) 
.50 (2) 1.00 (5) 
.29 (2) .80 (5) 
.29 (4) 1.00(11) 
.50 (4) .91(11) 
.67 (12)1.00 (7) 
.58(12) .29 (7) 



1.00(5) .60 (3) 
.40(5) .67 (3) 
.33(1)1.00 (7) 
.50(1) .86 (7) 

1.00(2) .33 (1) 

1.00(2) .33 (1) 
.75(12) .71 (5) 
.330-2)1.00 (5) 

1.00 (4) .62 (5) 
.75 (4) .80 (5) 
.95(19)100(11) 
.76(19). 82(11) 



.60 .40 
.44 .36 
.73 .71 
.69 .79 
.63 .60 
.58 .46 
.73 .39 
.74 .57 
.66 .43 
.74 .49 
.79 .50 
.65 .61 



Mean 



.65 



.35 



.83' 



.46 



.84 



.73 



.66 .52 



NOTE: 



a 



EnW - Engaged written 

NI - Non-engaged interim 



EnO - Engaged oral 

NW - Non-engaged waiting 



EnC 
NO 



Engaged covert 
Non-engaged off -task 



Observers 10 and 20 observed at WDRSD schools. 
Observers 30 and 40 observed at DMP schools. 



b Events coded by the expert. 

c -No Events were coded for this category by either 
the observer or expert. 
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agreement for the pair 10-90 was .97. Observer 20 misinterpreted the 
situation and coded manipulative rather than a game as discussed above. 
On the other occasion when games were coded as being observed, the 
expert observer (90) coded three events involving a game where the two 
observers did not code any games being used. Because of the low frequency 
of the use of games during the training session it is difficult to judge 

the facility of the observers tacnde this -category. —T his i s also -true 

for audio-visual aids (Table 6) . 

Printed materials (Table 7) on several occasions were coded by one 
of the observers but not by the expert as for the Thursday Grade 2 WDRSD 
class for pairs 30-80 and 40-80 and the Friday Grade 5 DMP classes for 
all pairs. On these occasions the number coded by the observer is small 
as can be deduced from the high marginal coefficients for not using 
printed materials. For the occasions where the expert coded printed 
materials being used, the marginal agreements are high (.92, .88, .85 and 
.95). Thus when printed materials were observed being used by the expert 
the agreement was reasonably high. This supports the conclusion that, 
along with the coefficients from the other material categories, the 
X observers had fairly high agreement on coding the categories coded most 
* frequently by the expert. But, with the exception of paper and pencil, 
the different kinds of materials were observed so Infrequently during 
the training session that it is difficult to assess the true capabilities 
of the observers to code these catMories from just the data collected 
during the training session except that there was high agreement that 
the materials were not used. 

There are differences among the agreement coefficients for the 
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agreement tor the .pair 10-90 was .96. Observer 20 misinterpreted the . 
situation and *oded manipulative rather than a game as discussed above. 
On the other occasion when a games were coded as being observed, the 
expert observer (90) coded three events involving a game where the two 
observers did not code any games being used. Because of the low frequency 
of the use of games during the training session it is difficult to judge 
Che facility of tite observers to code this category. . This is also true 
for audio-visual aids (Table 6). . 

printed materials (Table 7) on several occasions were coded by one 
of the observers but not by the expert as for the Thursday Grade 2 WDRSD 
class for pairs 30-80 and 40-80 and the Friday .Grade 5 DMP classes for 
all pairs. On these occasions the number'coded by the observer is small 
as can be deduced from the high marginal coefficients for not using 
printed materials. For the occasions where the expert coded printed 
materials being used, t^e marginal agreements are high (.92, .88, .85 
and .95). Thus when printed materials were observed being used by 
the expert the agreement was reasonably high. This supports the con- 
clusion that, along with the coefficients from the other material 
categories, the observers had fairly high agreement on coding the cate- 
gories coded most frequently by the expert. But, with the exception of 
paper an^pencil, the different kinds of materials were observed so 
■ infrequently during the training session that it is difficult to assess 
the true capabilities of the observers to code these categories from just 
the data collected" during the training session except that there was 

» 

high agreement that the materials were not used.. 
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There are differences among the agreement coefficients for the 
learner move categories (Table 8) that are partially accounted for by 
the frequency of % the number of events during which the category was 
coded. Engaged oral was very infrequently coded, less than 19% of the 
events for each class. The marginal agreements for this category are 
generally low and are unstable. Non-engaged waiting wds coded more 
frequently for some of the classes. The marginal agreements increased 
significantly from Thursday to Friday indicating improvement on coding 
this category. On Friday* the average marginal agreement for non-engaged 
waiting is .84. Non-engaged interim is a third category that was coded 
infrequently and had unstable marginal agreements. The agreements are 
reasonable for Thursday with an average marginal agreement of .76. For 
Friday the average marginal agreement is lower, .46, indicating some 
uncertainty of the observers in coding this category. The other three 
categories appear to be stable across days and acceptable. The average 
marginal agreements on Thursday and Friday for engaged written were .78 
and .65, for engaged cover, .82 and .83, and for non-engaged off-task .72 
and .73. For the set of categories over both days, the Cohen's K f s range 
from 127 to .79 with an average of .52. In general, the observers 
achieved an acceptable level of facility in coding three of the learner 
moves categories. They appeared to be more unstable in coding non-engaged 
interim. They improved sufficiently ton Friday in coding non-engaged 
waiting to indicate they have a grasp of that category. Engaged oral 
occurred too infrequently to make any conclusions about the capability 
of the observers to code this category. 

Table 9 lists the agreement coefficients, for two of the inter- 
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learner move categories (Table 8) that are partially accounted for by 
the frequency of the number of events during which the category was 
coded. Engaged oral was very infrequently coded, le6S than 10% of the 
events for each class. The marginal agreements for this category are 
generally low and are unstable. Non-engaged waiting was cod,ed more 
frequently for some of the classes. The marginal agreements increased 
significantly from Thursday to Friday indicating improvement on coding 
this category. On Friday the average marginal agreement for non-engaged 
waiting is .84. Non-engaged interim is a third category that was coded 
infrequently and had unstable marginal agreements. The agreements are 
reasonable for Thursday with an average marginal agreement of .76. For 
Friday the average marginal agreement is lower, .46, indicating ,«ome 
uncertainty of the observers in coding this category. The other three 
categories appear to be stable across days and acceptable. The average 
marginal agreements on Thursday and Friday for engaged written were .78 
and .65, for engaged covert, .82 and .83, and for non-engaged off-task, ' - 
.72 and .73. For the set of categories over both days, the Cohen f te. k's 
range from .27 to- .79 with an average of .52. In general, the observers 
achieved an acceptable level of facility in coding three of the learner 
moves categories. They appeared to be more unstable in coding non- 
engaged interim. They improved sufficiently on Friday in coding non- 
engaged waiting to indicate they have a grasp of that category. Engaged 
oral occurred too infrequently to make any conclusions about the capability 
of the observers to code this category. j 

Table 9 lists the agreement coefficients for two of the interaction 
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categories,, target to student and teacher to group. These two 
categories were the only ones which were consistently observed across 
classes. The number of events coded by the expert for target to student 
was very* small. iue marginal agreements are unstable reflecting t;he 
low number of occurrences coded. There is some indication that observers 
tended to code more than the expert interactions of the target student 
talking to another student as indicated by the lack of , agreement, for 
example, on Friday for the Grade 5 DMP class. On nine of the twenty 
occasions the marginal agreement for the category was .75 or above. These 
occasions included the two classes where the expert coded the greatest 
number of events involving this category, Thursday for Grade 2 DMP with 



pairs 30-90 and 40-90 and on Friday for Grade 2 WDRSD with pairs 30-80 
and 40-80. 

The category teacher to group was coded more frequently by the 
experts than the observers with the marginal agreement averaging .77 for 
Thursday and .45 on Friday.^ On the occasions where the expert coded the 
category for at least 10% of the events, the range of marginal agreement 
was above .60 except for one paitf. The marginal agreements for not coding 
any interactions are high ranging from .71 to 1.00 with an average of .91. 
Thus observers were consistent in agfeeing when interactions did not occur. 
Observers were less consistent in coding positive occurrences of the 
categories target to student and teacher to group. On most occasions 
these categories occurred infrequently which -contributed to the instability 
of the agreement coefficient.* It also is difficult to interpret the 
coefficients for the set of interaction categories because of the low 

4 

frequency of the categories. 
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TABLE 9 



Summary of tM Category Marginal, Overall Marginal 
and Cohen's Kappa Observer Agreement Coefficients for 
the DMP and WDRSD Observers with the Expert Observers 

Training Session: January 12 and 13, 1978 



Interactions Category 



Observer 
Pair 

Obs a Expert 



Curriculum 
and 
Grade 



Total 
Events 
Observed 



Category Marginal 

Target to Teacher to 
Student Group 



None of 
the above 



Overall 
Marginal 



Thursday, January 12, 1978 



Cohen ' s - 
Kappq 
(K) 



10. 


\ 90 


WDRSD 


2 


54 


.50 


(2) b 


.46 (5) 


.71 


(45) 


.39 


.27 


20 


90 


WDRSD 


2 


' 54 


1.00 


(2) 


.80 (5) 


.96 


(45) 


.65 


.49 


30 


80 


WDRSD 


2 


60 


.75 


(3) 


.73 (11) 


.96 


(43) 


.30 


.73 


AO 


80 


WDRSD 


2 ' 


60 


.60 


(3) 


.92 (11) 


.88 


(43) 


.56 


.67 


10 


80 


DMP 


2 


. 84 


.17 


(1) 


.50 (8) 


.93 


(56) 


.47 


.32 


20 


80 


DMP 


2 


84 


,50 


(1) 


.80 (8) 


.84 


(56) 


.29 


.32 


30 


90' 


DMP 


2 


78 


1.00 


(6) 


.92 (13) 


1.00 


(55) 


.49 


.54 


AO 


90 


DMP 


2 


78 


.83 


(6) 


1.00 (13) 


1.00 


(55) 


.81 


.73 




Mean 








.67 




.77 


.91 




.50 


.51 



10 
20 
30 
40 
10 
20 
30 
40 
10 
20 
30 
40 



Friday., January 13, 1978 



50 


DMP 


5 


30 


.50 


(1) 


90 


\ DMP 


5 


30 


1.00 


(1) 


80 


\ DMP 


5 


30 


- c 


(0) 


80 


DMP 


5 


30 


.00 


(0) 


90 


WDRSD 


2 


54 


.00 


(0) 


90 


WDRSD 


2 


54 




(0) 


80 


WDRSD 


2 


54 


.75 


(4) 


80 


WDRSD 


2 


54 


1.00 


(4) 


80 


DMP*, 


2 


78 


1.00 


(2) 


80 


DMP 


2 


78 


1.00 


(2) 


90 


DMP 


2 


84 


.33 


(3) 


90 


DMP 


2 


84 * 


.50 


(3) 



Mean 



.61 



- (0) 


.93 


(26) 


.29 


.12 


- (0) 


.93 


(26) 


.32 


.30 


.67 (3) 


.96 


(26) 


.41 


.53 


.60 (3) 


.88 


(26) 


.30 


.48 


.73 (8) , 


.89 


(45) 


.52 


.32 


.13 (8) 


.86 


(45) 


.25 


.14 


0.00 (0) 


.96 


(47) 


.53 


.31 


- <0) 


.9* 


(47) 


.40 


.67 


.62 (21) 


.86 


(49) 


.35 


.46 


.88 (21) 


.86 


(49) 


.61 


.64 


- (0) 


.92 


(75) 


.38 


'.18 


0.00 (0) 


•95 


(75) 


.35 


.36 


.45 


.92 




.39 


.38 



a Observers 10 and 20 observed at WDRSD schools. 
Observers 30 and 40 observed at DMP schools. 

b Events coded by the expert. 

c -No Events were coded for this category by either 
the observer or expert. 
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Summarizing the results of the analyses of the marginal agreements, 
for most categories the marginal agreements between the observer and the 
expert are consistently hi&h. A few categories were coded very infre- 
quently making it difficult to assess the ability of the observers in 
coding these. These categories are pair, small group, engaged oral and 
most of the materials and interactions categories. The marginal coefficients 
for the non-engaged interim category varied by observer pair and from class 
to class. The coding of this category is more unstable than the others 
which may be related to how it is defined. 

Generalizability of the Categories . The generalizability coefficients 
from the training session for each observer trio and for each category are 
given in Table 10. These can be interpreted as an indication of the amount 
of variance due to the class when considering the interaction between the 

observers and tt\e class. A high coef f icier) t indicates that most of the • 

< 

variance is due to the class and not to the difference among the observers. 
A low coefficient indicates that there is some interaction among the 
observers and the class. That, is, observers coded the category differently 
for different classes. 

The coefficients for most of the categories are high, above .80. 
Categories /with consistently lower coefficients are pair,, engaged -oral, 
non-engaged interim, and non-engaged waiting. Engaged-written has very 
low coefficients for the sessions with observer trios 10,20,80 and 30, AO, 
90 but has high coefficients for the other three classes. Zero coefficients 
for some of the categories (engaged oral and non-engaged interim and waiting) 
indicate that there was no variance due to class. Thus the classes observed 

46 
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TABLE 10 

Summary Generalizability Coefficients 
Training Session 



Observation 
Category 


3 classes 


- 2 days 


2 classes - 


■ 2 days 




Observer Trio 


Observer 


Trio 




10x20x90 


30x40x80 


10x20x80 


30x40x90 


• 


i 

J. • 


Rpl f — oft p fid 


.96 


.99 


.49 


.82 




£ • 


Ofhpr— nAced 


.99 


.97 


.83 


.77 


• 


•J • 


nptieral Content 

WW Aft W A* Cft •!» W*ft V Wftft l» 


.36 


.69 


.98 


.90 




A 


Individual 


.95 


.99 


.96 


.64 






Pair 

* CI X I* 


.70 


.76 


.49 


.33 




6. 


Small Group 


,94 


.57 


.95 


.99 




7. 


Large Group 


.99 


.98 


.95 


.98 




8. • 


General Content 


.78 


.69 


.98 


.40 




9. 


Engaged-Written 


.86 


.91 


.05 


.10 




10. 


Engaged-Oral 


.81 


.61 


.00 


.00 




11. 


Engaged-Covert 


.81 


.94 


.46 


.76 




12. 


Non Engaged-Interim 


.97 


.37 


.00 


.45 




13. 


Non Engaged-Waiting , 


.93 


.63 


.00 


.00 




14. 


Non Engaged-Off Task 


.89 


.71 


.31 


.89 




15. 


General Content 


.36 


.69 


.98 


.40 




16. 


Paper /pencil Materials 


.98 


.99 


.99 


.94 




17. 


Total Engagement 


.90 


.93 


.59 


.95 


r 


18. 


Total Non Engagement 


.95 ' 


.85 


.04 


.96 
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were very similar on these categories. For the categories of pair 
and engaged written, the low non-zero coefficients indicate that the 
variance due to the interaction between the observer and class was high s 
compared to the variance due to the class. On these categories there is 
more disagreement among the observers. 

Generalizability coefficients are given for the sum of the three 
engaged categories (Total' Engagement) and for the three non-engaged 
categories (Total Non-engagement). With the exception of one observer 
trio, the coefficients /or these tW composite categories are high. 



Observers in some cases disagreed as to how the students were engaged or 
non-engaged, but they did agree that the students were engaged or non- 



categories the observers demonstrated that they achieved a high degree, 
of competency in coding the category. In coding the categories of pair, 
non-engaged interim and non-engaged waiting the observers were less con- 
sistent. The agreement among observers on these categories need to be 
checked carefully during the on-site visits and retraining sessions. A 
few categories (small group and engaged oral) were rarely observed by 
/ either the expert or the observers during the training session making 
it impossible to assess the true ability of the observers to code these., 
categories except that there was high agreement that the categories did 
not occur. 




engaged . 



Summary of Training Session Obs er ver Agreement . On many of the 
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Retraining Session 

The summaries of the category marginals, overall marginals, and the 
Cohen's K for the DMP and WDRSD retraining session are given in Tables 
11 through 20, Table 21 gives the summary of means of the category 
marginals for both the training and retraining sessions. The summary of 
- the means of the Cohen's K for both sessions are given in Table 22. The 
generalizability coefficients for each category for the retraining session 
are given in Table -23. The contingency tables showing the frequencies for 
each category by observer pair for the retraining session are included in 
Appendix B. 

There is still variation in the marginal agreements for categories 
from class to class. Often the lower marginal agreements correspond to a 
low frequency of positive, occurrences observed by the. expert. For example, 
in Table 12 for Thursday Grade 2 WDRSD, the lower marginal coefficients 
(.57 and .56) occurred when the expert only coded 7 or 5 events as 
individual. In Table 11 for other paced, the lower marginals of .00 and 
.43 on Thursday occurred when 2 and 7 events, respectively, were coded by 
the expert as being other paced. It appears that the observers are coding 
the main activity with accuracy, but if there is some deviation, such as 
one student working individually while the other students are working in 
a large group, this point is missed on occasion by some of the observers. 

The means of the category marginal (Table 21) indicate that on the 
categories of pair, games, printed materials, engaged -writing, engaged-oral, 
non-engaged off-task, and teacher to group interactions the observers 
improved in /heir ability to code these after doing seven observations. 
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The mean marginals were down considerably for manipulatives and slightly 
for other-paced and target to student interactions. Manipulatives were 
not used as much during the classes observed for the retraining session 
(Table 14) as they were during the training session (Table 4). Again 
observers agreed closely on what materials were used over a long period 
of time, but were more inconsistent in agreeing when materials were used 
only on a few occasions. The mean of the marginals for the other categories 
remained approximately the same. 

1 — <fonsidering each set of categories and the means of the Cohen's K, 
paper afSa^pencil materials, printed materials, and learner moves generally 
had means higher for the retracing session than for the training session 
(Table 22). The only category with a distinct decrease in the means of 
the Cohen's K is manipulatives. The Cohen's K can be interpreted as the 
percent of agreement beyond chance agreement. The k's above .70 are 
considered to be reasonably high. 

Considering both the*data from the training and retraining sessions 
there is reasonable agreement on the following categories: 

Self paced 

Other paced 

Individual 

Large Group 

Paper /pencil 

Engaged-written 

Engaged-covert 

Non-engaged off -task 
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The agreement on the category engaged-oral was low for the training 
sessions, but improved to an acceptable level for the retraining session. 
Students only occasionally would talk during the classes observed so that 
the category of .engaged -oral had very low frequencies. The class with 
the highest frequency of positive 'occurrences of engaged-oral for the 
retraining session (Table 19) had 8 of 90 events coded by the expert as 
having engaged-oral. 

The categories with consistently low observer agreement are: 

Pair 

Small group 

Printed materials 

Non-engaged interim 

Non-engaged waiting 

Target to student interactions 

Teacher to group interactions 
For the categories of pair and small group for the retraining sessions 
(Table 12), agreement was reasonable if, for either of the groupings, 
the grouping occurred on more than two events. Disregarding the mai#Lnal 
agreements for pair when the expert coded pairs for two or less events, 
the average marginal agreement was .70 and .76 for the two days. On no 
occasion in the retraining session did an observer code a pair when the 
expert did not code any pairs. For the five observer pairs where the 
expert observed more than two small groups, the average marginal agreement 
was .90. On only one of the occasions, Grade 2 DMP Friday observers 10-20-90, 
did the observers code a small group when the expert did not observe any 
For these two categories, it appears that the agreement was high or 
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reasonable when pairs and groups were used more than just on occasion 
during the period. The observers were not as sensitive when two or 
three students would form a group, discuss some work and then continue 
with their own individual work. These "informal" pairs and small groups ' 
were more difficult for the observers to identify. 

For printed materials (Table 17) the agreement was high when printed 
materials were not used. The disagreement on when printed materials were 
used is^ipainly due to deciding if the printed materials were part of t^e 
on-gOing activity. The observers were to code printed materials if the 
materials were on the desk and used at sometime during the on-going activity, 
The student did not have to be reading from the printed material at the 
moment observed for printed materials to be coded as part of the on-going 
activity. Instead observers generally coded printed materials when they 
were actually being used. This is also true for the other materials 
categories of games, audio-visual and manipulatives. However, these 
materials were used very infrequently making it difficult to get any 
true measure of observer agreement. 

, The two non-engaged categories of interim and waiting are similar 
to pair and small group categories. On occasions where the expert 
observed two or more events with one of these categories, there was 
reasonable agreement. The average marginal agreements for each of the 
two days, when only counting occasions with more than two events coded 
by the expert, were .80 and .74 for interim and .75 and .69 for waiting. 
Again the observers agreed reasonably well if the category occurred on 
more than two events. Disagreement on these categories generally occurred 
when there was little time spent in interim or waiting activities. 
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Interactions occurred infrequently during the classes (Table 20). 
Marginal agreements are reported for the two interactions that were coded 
the most, target to student and teacher to group. For both of these 
interactions the number of positive events observed by the expert was 
generally 20% or less of the total number of events coded. The marginal 
agreements are high for observer pairs coding when there were no inter- 
actions. The marginal agreements, however, are low on target to student 
interaction. For teacher to group interactions the agreements are 
reasonably high on one day (Friday), but lower on the other. The average 
marginal agreements are similar to those for the training session with 
a similar variation between the two days. Observer agreement is acceptable 
in coding teacher to group interactions since agreement is generally high 
when there were a significant number of interactions, more than 10% of 
the events. ^k£o the agreement increased on the second day of the training 
session indicating improvement of the ability in observing this category. 
The frequency of occurrence of target to students was so small that it is 
difficult to measure the true capability of the observers in coding this 
category except that there is high agreement in not coding the category. 

Generallzablllty Coefficients . I The generallzablllty coefficients 
computed across the three classes fqr both days of the retraining session 
for each observer group are given in Table 23. The coefficients are 
calculated treating day as a fixed effect. That is, the assumption was 
made that variations in the categories from day to day are expected and are 
not treated as error. Most of the coefficients are extremely high, above 
.90, indicating that the variation among the observers was very low in 
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relationship to the variation among the classes. In some cases the 
coefficient is zero, general content for observers 10, 20, 90 and 
engaged-oral for observers 40, 80. This indicates that there was very 
little variation among the classes for the category. In the case of 
general content one observer group had .00 and the other had a coefficient 
of .96. This indicates that when there was enough variation among the 
classes on general content the observers agreed highly on this category. 
For engaged-oral, both coefficients -are low. Thus there is disagreement 
on this category even though there is little variation among classes- 
partly because of the low frequency of the category. The only other 
category that is low for both observer groups is pair. There is some 
disagreement among observers across classes in ^coding the use of pairs. 

Other categories have coefficients that vary between the two 
observation groups — engaged-covert, total engagement, non-engaged interim, 
and non-engaged off task. The observers in one of the groups varied some- 
what on these categories in realtionship to the variance among classes. 
This indicates that these categories are more unstable than the others 
but still reasonably high with coefficient values of .60 or above. Total 
engagement for the group 10,20,90 has a coefficinet of .46 which is 
fairly low. This probably is associated with the variation among observers 
in coding engaged-oral since the coefficients for the other two engaged 
categories are high. 

Retraining Session Summary . On most of the categories the agreement 
among observers in coding the categories is high as indicated by the high 
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marginal agreements and generalizability coefficients, The categories 
with questionable agreement are generally those which were observed 
infrequently. The coding of pairs was low as indicated by both the 
marginal agreements and the generalizability- coefficients. Observers * 
improved from the training session to the retaining session in coding 
engaged-oral, but observers still varied greatly on the category as indicated 
by the low generalizability coefficients. Non-enga^ged interim and non- 
engaged waiting both had low marginal agreements but had relatively" high 
generalizability coefficients. Observers disagree somewhat in coding one 
class on these categories but the agreement is sufficient enough to 
distinguish the use of these categories among classes. Frequent use of 
small groups is coded with high agreement. An occasional use of small 
groups where three or four students come together briefly to work on a 
problem and then continue with their individual work is more difficult 
for the observers to identify. Overall, the observers code small groups . 
with enough accuracy to distinguish among classes as indicated by the high 
generalizability coefficients for this category. Observers have high • 
agreement in instances when interactions do not occur. There is dis- 
agreement in observing particular types of interactions. The different 
kinds of interactions occur so infrequently that it is difficult to 
assess the agreement among the observers when observing in actual class- 
room situations. Similarly the observance of the use of most of the 
kinds of materials was so infrequent that it is impossible with the data 
collected to assess what is the agreement among observers except that 
they agreed highly that the materials were not used. 
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Summary of the Category Marginal, Overall Marginal 
and Cohen's Kappa Observer Agreement Coefficients for 
the DMP and WDRSD Observers with the Expert Observers 

Retraining Session: March 16 and 17, 1978 

Pace 



Observer Curriculum Total 

Pair and Events 

Obs a Expert Grade Obse rved 



Category Marginal 



Self 



Other 



No Pace 



Overall Cohen's 
Marginal Kappa K 



Thursday, March 16, 1978 



10 


90 


. DMP 5 


90 


."99 


(82) c 


20 


90 






.99 


(82) 


30 


80 


DMP 5 








40 


80 


DMP 5 


90 


.88 


(29) 


10 v 


90^ 


WDRSD 2 ' 
WDRSD 2 V , 


66 


' .88 


(8) 


20 \ 


90 


66 


1.00 


(8) 


30 


80 


WDR$D 2 






(5) 


40 


v80 
90 


WDRSD 2 


60 


.56 


10 


DMP 2 


<•> 


.96 


(70) 


20 


. '90 


DMP 2 


84 


.96 


(70) 


30 


v 80 


DMP 2 








40 


' 80 


DMP 2 


78 


".85 


(44) 



.60 (3) 
.00 (3) 

.96 (49) 
.90 (46) 
.46 (46) 

.89 (37) 
.43 (7) 
.78 (7) 

.83 (30) 



.80 (5) 
1.00 (5) 

.50 (12) 
.67 (12) 
.8>H2) 

1.00 (18) 
.88 (7) 
.88 </) 

.25 (4) 



.80 
.57 

.78 
.82 
.93 

.82 
.75 
.87 

.64 



Mean 



.87 



.82 



.74 



.81 



-f- 



\ 

a Observers 10 and 20^bserve<^atWDRSD schools. 
Observers 30 and JrO/ observed atTDMP schools. 



56 
64 

,76 
.79 
.76 

.88 
.78 
.66 

.69 





Mean 




1 


.90 




.70 


.76 


.78 


.65 










— 1 'j 

Friday, 


March 


17, 1978 








10 


■ '. — t — 

90 


DMP 5 


90 


.97* 


(75) 


1.00 (2) 


.85 (13) 


.94 


.75 . 


20 


90. 


DMP 5 


90 


.97 


(75) 


.50 (2) 


.92,(13) 


.80 


.83 


30 




DMP 5 


90 


' ' .76 


(38) 


.82 (50) 


.00 (2) 


.53 


.73 


'40 


80 ^ 


DMP 5 


90 


.95 


(38) 


.96 (50) 


1.00 (2) 


.97 


.83 


10 


90 


WDRSD 2 


66 


.78 


(25) 


.76' (33) . 


.89 (8) 


.81 


.73 


20 


90 


WDRSD 2 


66 


■ •74 


(25) 


.73 (33) 


1.00 (8) 


.82 


.68 


30 


80 


WDRSD 2 


60 


.92 


(24) 


1.00 (32) 


.50 (4* 


.81 . 


.82 


40 


80 , 


WDRSD 2 


N 60 


.92 


(24) 


1.00 (32) 


.50' (4) 


.81 


.94 


10 


90 


DMP 2 


84 


.85 


(35) 


.86 (42) 


1.00 (7) 


.99 


.71 , 


20 


90 


DMP 2 


8'4> 


.88 


(35) 


.83 (42) 


.78 (7) 


.83 


.69 


30 


80 


DMP 2 


84 


.77 


(62) 


.63 (19) 


.50 (3) 


.64 


.62 


40 


80, 


DMP 2 


84 


.90 


(62) . 


.76 (19) 


1.00 (3) 


.89 


.84 



• 76 



b Observer 30 did not attend the first day of the retraining. 



c Events coded by the expert. 
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TABLE 12 > 

Sunmary of the Category Marginal, Overall Marginal 
and Cohen's Kappa Ob-server Agreement Coefficients for £ 
the DMP and WDRSD Observers with the Expert Observers 

* 

Retraining Session: March 16 and 17, 1978 

. ■ * i 

Grouping \ 



Observer Curriculum 

Pair and 
Obs a L xpe rt ' firaclc 



10 

20 
30 
40 
10 
20 
30 
40 
10 
20 
30 
AO 



Total 

Events Ind 
Observed 



Pair 



Category Material 

Small Large 
Group Croup 



No 
Group 



Overall 
Marginal 



Cohen's 
Kappa < 



$0 
90 
80 
80 
90 
90 
00 
80 
90 
90 
80 
80 



i»'.r 5 

DM' 1 5 
DMT 5 
DM? 5 
WDRSD 2 
WDRSD 2 
WDRSD 2 
WDRSD 2 
DMP 2 
DM? 2 
DMP 2 
DMP 2 



Moan 



T hursday^ March 16, 197b 



10 


90 


DMP 5 


90 


.92(66) c 


.53(17) 


20 


90 


DM? 5 


90 


.94(06) 


• 7K17) 


30 * 


80 


DM? 5 








40 


80 


DM? 5 


90 


.88(28) 


.50 (2) 


10 


90 


WDRSD 2 


66 


.86 (7) 


.67 (2) 


20 


90 


WDRSD 2 


66 


.57 (7) 


.50 (2) 


30 


80 


WDRSD 2 








40 


80 


wd:;sd 2 


60 


.56 (5) 


- (0) 


10 


90 


DMP 2 


84 


.88(57) 


.64(H) 


20 


90 


DM? 2 


84 


.93(57) 


.92(11) 


30 


80 


DMP 2 






.2? (2) 


40 


80 


DMP 2 


78 


.93(4?.) 




Mean 






•> 


.60 



90 
90 
90 
90 
66 
"66 
60 
60 

'84 
84 



.92(62) 
.98(62) 
.74(31) 
.78(31) 
.75(18) 
.86(18) 
.84(21) 
.91(21) 
.92(33) 
.97(33) 
.92(48) 



84 1.00(48) 



.88 



.40 r(2) 

.50 (2) 

- (0) 

- (0) 

- (0)- 

.89 (37) 
.00 (2) 
,00 (2) 

.33 (12) 



' -<*(0) 
- (0) 

.94(48) 
.92(45) 
.94(45) 

- (0) 
,.57 (7) 
.43 (7) 

.86(18) 



.80 (5) 
.71 (5) 

.50 (12) 
.6/ (12) 
.81 (12) 

1.00 (18) 
.88 (7). 
.88 (7) 

.25 (4) 



.35 



78 



72 



Friday. March 17. 1978 



.86 (7) 
.78 (7) 
.62 (8) 
.25 (8)* 
.82 (11) 
.85 (ID 
1.00 (3) 
1.00 (3) 
• .42 (5) 
.83 (5) 
.92 (12) 
.75 (12) 



.75 (8) 
.88 (8) 
*.00 (2)f 
.00 (2) 
.50 (1) 
.25 (1) 

- (0) 

- (0) 
.00 (0) 
.00 (0) 

1.00 (3) 
1.00 (3) 



.00 (0) 
.87 v 47) 
.98 (47) 
. 79 (28) 
.71 (28) 
.94 (32) 
1.00 (32) 
.69 (39) 
.87 (39) 
.78 (18) 
.86 (18) 



.85 (13) 
.92 (13) 



.00 
1.00 

.89 
1.00 

.30 

.50 
1.00 

.78 
1.00 
1.00 



(2) 
(2) 
(8) 
(8) 
(4) 
(4) 
(7) 
(7) 
(3) 
(3) 



.66 
.72 

.70 
.78 
.71 

,*82 
.59 
.63 

.53 



.66 



.b4 
.71 
.45 
.60 
.75 
.73 
.82 
.85 
.60 
.69 
.92 
.92 



.76 



.44 



.77 



.79 



.74 



.49 
,47 

.75 
.80 
.74 

.88 
.66 
.65 

.56 



,67 



.68 
.67 
.72 
.69 
.70 
.65 
.89 
.94 
.66 
.72 
.77 
.74 



.74 



a Observers 10 and 20 observed at WDRSD schools. 
Observers 30 and 40 observed at DMP schools • 

b Observer 30 did not attend first day of the 
retraining session. 

ERIC 
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c Events coded by the expert. 

d No Events were coded for thij 
category by either the expert 
or the observer. 



TABLE 13 

s 

Summary of the Category Marginal, Overall Marginal 
and Cohen's Kappa Observer Agreement Coefficients for 
the DMP and WDRSD Observers with the Expert Observers 

Retraining Session: March 16 and 17, 1978 
Taper/Pencil 
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Observer Curriculum Total 
pair • and . Events 
0bs a ' Expert Grade Observed 



Category Marginal 



10 
20 
30 
AO 
10 
20 
30 
AO 
10 
20 
30 
AO 



Paper /pencil Paper /pencil 
used not used 



Other 



Overall Cohen's 
Marginal Kappa K 



Thursday. March 16, 1978 



90 
90 
80 
80 
90 
90 
80 
80 
90 
90 
80 
80 



DMP 5 
DMP 5 
DMP 5 
DMP 5 
WQRSD 2 
WDRSD 2 
WDRSD 2 
WDRSD 2 
DMP 2 
DMP 2 
DMP 2 
DMP 2 



90 
90 
90 
90 
66 
66 
60 
60 
84 
8A 
8A 
8A 



.96 (77) 
1.00 (77) 
.88 (76) 
.93 (76) 
.58 .(41) 
.98 (41) 
.96. (51) 
.94 (51) 
.89 (47) 
.94 (47) 
.96 (66) 
.97 (66) 



.00 (0) 
.00 (0) 
.33 (12) 
.71 (12) 
.52 (17) 
.94 (17) 
1.00 (5) 
.80 (5) 
.86 (30) 
.83 (30) 
.83 (15) 
.87 (15) 



.84 (13) 
.92 (13) 
.00 (2) 



1.00 
.89 

1.00 
.50 
.50 

1.00 
.78 

1.00 

1.00 



(2) 
(8) 
(8) 
(4) 
(4) 
(7) 
(7) 
(3) 
(3) 



.60 
.64 
.41 
.88 
.66 
.97 
,.82 
.75 
.92 
.85 
.93 
.95 



10 
20 


90 
90 


DOT 5 
DMP 5 


90 
9 _°b 


.76 
1.00 


(82) c 
(82) 


.12 (3) 
.33 (3) 


.80 (5) 
.71 (5) 


.56 
.68 


.29 

.66 - 


30 
AO 
10 
20 


80 
80 
90 
90 


DM1' 5 
DMP 5 
.WDRSD . 2 
WDRSD 2 


90 
66 
66 


.85 
.83 
.80 


(60) 
(10) 
(10) 


.54 (18) 
.96 (44) 
.92 (44) 


.50 (12) 
.67 (12) 
.83 (12) 


.,63 
.82 
.85 


.59 
.78 
.8A 


30 
AO 
10 
20 


80 
80 
90 
90 


WDRSD 2 ■ 

WDRSD 2 
DMP 2 
DMP 2_ 


60 
8A 
8A 


.7A 
.99 
.99 


(25) 
(77) 
(77) 


.47 (17) 

- d (0) 

- (0) 


1.00 (18) 
.88 (7) 
.88 (7) 


.74 
.93 
.93 


.76 
.93 
.93 


30 
AO 


80 
80 


DMP 2 
DMP 2 


■ 78 


.98 


(47) 


.93 (27) 


.25 (4) 


.72 


.87 




Mean 






.88 




.61 


.72 


.76 


.7A 








' ■ 1 


Friday, 


March 17, 


1978 









.72 
.91 
.23 
.82 
.59 
.80 
.86 
.78 
.75 
.72 
.90 
.93 



Mean 



.92 



.6A 



- .78 



.78 



.75 



ERIC 



a Observers 10 and 20 observed at WDRSD schools. 
Observers 30 and AO observed at DMP schools. 

i 

b Observer 30 did not attend first day of the retraining session, 
c Events coded by the expert. s 

d No events were coded for this category by either the expert or the observer. 
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TABLE 14 



Summary of the Category Marginal, Overall Marginal 
and Cohen's Kappa Observer Agreement Coefficients for 
* the DMP and WDRSD Observers with the Expert Observers 

Retraining Session: March 16 and 17, 1978 
Manipulative Material Category 



Observer Curriculum 

Pair and 
Ob s a Expert Grade 



Total : 

Events Manipulative 
Observed Used 



Category Marginal 



Manipulative 
Not Used 



Other 



Overall Coh%ft's 
Marginal Kappa K. 



Th ursday, March 16, 1978 



10 
20 


90 
90 


DMP 5 
DMP 5 


90 
90 b 


.25 (4) c s> 
.00 (4) 


.95 (81) 
.98 (81) 


.80 (5) 
.71 (5) 


.67 
.56 


.55 
.60 


30 
40 


80 

4 80 


DMP 5 
DMP 5 


P 

90 


.41 (15) 


.75 (63) 


.50 (12) 


.55 


.52 


10 


90 


WDRSD 2 


66 












20 


90 


WDRSD 2 


66 












30 
40 


80 
80 


WDRSD 2 
WDRSD 2 


60 


.11 (9) 


.86 (65) 


.25 (4) 


.40 


.10 


10 


90 


DMP 2 


84 












20 


90 


DMP 2 


84 












30 


80 


DMP 2 














40 


80 


DMP 2 


78 














Mean 






.19 


.88 


s -56 


.54 


.44 










Friday, March 17 


, 1978 








10 


90 


DMP 5 


90 


.00 (0) 


1.00 (77) 


.85 (13) 


.62 


.82 


20 
30 
40 


90 
80 
80 


DMP 5 
DMP 5 
DMP 5 


90 
90 
90 


.48 (12) 
.67 (12) 


.86 (76) 
.95 (76) 


.00 (2) 
1.00 (2) 


.44 
.87 


.43 
.72 


10 


90 


WDRSD 2 


66 












20 
30 
40 
10 


- 90 
80 
80 
90 


WDRSD 2 
WDRSD 2 
WDRSD 2 
DMP 2 


66 
60 
60 
84 


.98 (56) 
.80 (56) 
.00 (0} 


.00 (0) 
.00 (0) 
.99 (77) 


.50 (4) . 
.50 (4) 
1.00 (7) 


.49 
.44 
.66 


.41 
.27 

.49 ' 


20 
30 
40 


90 
80 
80 


DMP 2 
DMP 2 
DMP 2 


84 
84 
'84 


.00 (1) 
1.00 (1) 


.99 (80) 
1.00 (80) 


1.00 (3) 
1.00 (3) 


.66 
1.00 


.85 
1.00 




Mean 






.49 


.72 


.73 


.65 


.63 



b 
c 



Observers 10 and 20 observed at WDRSD schools. 
Observers 30 and 40 observed at DMP schools. 

Observer 30 did not attend Thursday of the Retraining session. 
Events coded by the expert. 



ERIC 



KQ 
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TABLE 15 



> Summary of the Category Marginal, Overall Marginal 
and 'Cohen's Kappa Observer Agreement Coefficients for 
the DMP and WDRSD Observers with the Expert Observers 

Retraining Session: March 16 and 17," 1978 

Game 




^ Observer 
Pair 
Obs a • 



Curriculum Total 
and Events 
Ex pert Grade Ob s erved 



Overall Cohen 1 
Marginal Kappa 



Thursday | March 16, 1978 



10 


90 


DMP 5 


90 


20 


90 


DMP 5 




30 


80 , 


DMP 5 




AO 


80 


DMP 5 


90 


10 


90 


WDRSD 2 


66* 


20 


90 


WDRSD 2 


66' 


30 


80 , 


WDRSD 2 




40 


80 


WDRSD 2 


60 


10 


90 


DMP 2 


84 


20 


90 


DMP 2 


84 


30 


80 


DMP P 




40 


80 


DMP 2 


78 


Mean 



.00 (4) c 



.88 (74) 



.50 (12) .46 



.50 



.00 



.88 



.50 



March 17, 1978 



10 


90 


DMP 5 


90 


20 


90 


DMP 5 


90 


30 


80 


DMP 5 


90 


40 


80 


DMP 5 


90 


10 


90 


WDRSD 2 


66 


20 


90 


WDRSD 2 


66 - 


30 


80 


WDRSD 2 


60 


40 


80 


WDRSD 2 


60 


10 


90 


DMP 2 


84 


20 


90 


, DMP 2 


-84 


30 


80 


DMP 2 • 


84 


40 


80 


DMP 2 


\ 84 



.00 (34) 

.00 (34) 

.80 (8) 

.75 (8) 



.56 (43) 

.57 (43) 

.97 (73) 

.97 (73) 



1.00 (7) 
.78 (7) 
1.00 (3) 
1.00 (3) 



.46 




.52 
.45 
.92 
.91 



.11 
.13 
.91 
.89 



Mean 



.39 



.77 



.94 



.70 



.51 



a Observers 10 and 20 observed at WDRSD schools. 
Observers 30 and 40 observed at DMP. schools. 

b Observer did not attend first day of the retraining session. 

c Events coded by the expert. 



60 



52 



TABLE 16 



\ 



Summary of the Category Marginal, Ovoxall Marginal 
and Cohen's Kappa Observer Agreement Coefficients for 
the DMP and WDRSD Observers with the Expert Observers 

Retraining Session; March 16 and 17, 1978 
Audio-Visual Aid 



Observer Curriculum Total 

Pair. and Events 

Obs a Expert Grade Observed^ 



Category Marginal 



AV Aid 
Used 



AV Aid 
Not Used 



Other 



Overall Cohen's 
Marginal Kappa K. 



Thursday, March 16, 1978_ 



10 

20 

30, 

AO 

10 

20 

30 

AO 

10 

20 

30 

AO 



90 
90 
80 
80 
90 
90 
80 
80 
90 
SO 
80 
80 



DMP 5 

DMP 5 

DMP 5 

DMP 5 
WDRSD 2 
WDRSD 2 
WDRSD 2 
WDRSD 2 

DMP 2 

DMP 2 

DMP 2 ' 

DMP 2 



90 

90 

..b 

90 
66 
66 

60 
8A 

\ 8A 

78 



Mean 



10 


' 90 


DMP 5 


90 


20 


90 


DMP 5 


90 


30 


80 


DMP 5 


90 


AO 


80 


DMP 5 


90 


10 


90 


WDRSD 2 


66 


20 


90 


WDRSD 2 


66 


30 


80 


WDRSD 2 


60 


AO 


80 


WDRSD 2 


60 


10 


90 


DMP 2 


8A 


20 


90 


DMP 2 ' 


8A 


30 


80 


DMP 2 


84 


AO 


80 


DMP 2 


8A 



Friday, March 17. 1978 



.00 (0) 



,95 (58) 1.00 (8) 



.65 



.82 



9 

ERIC 



Mean - 00 - * 95 1,00 ,65 ,82 



a Observers 10 and 20 observed at*WDRSD schools. 
Observers 30 -and - 40 ob serv e d at DMP schools* — 



b Observer did not attend first day of the retraining session, 
c Events coded by the expert. 
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TABLE 17 



Summary of the Category Marginal, Overall Marginal 
and Cohen's Kappa Observer Agreement Coefficients for 
the DMP and WDRSD Observers with the Expert Observers 

0 

Retraining Session: March 16 and 17, 1978 
Printed Materials 



Observer Curriculum Total 

Pair and Events 

Obs a Exper t Grade Obs erved 



10 
20 
30 
40 
10 
20 
30 
40 
10 
20 
30 
40 



Printed 
Materials 

Used 



Category Marginal 



Printed, 
Materials 

Not. Used 



Other 



Overall Cohen's 
Marginal Kappa K 



Thursday, March 16, 1978 



90 
90 
80 
80 
90 
90 
80 
80 
90 
90 
80 
80 



DMP 5 
DMP 5 
DMP 5 
DMP 5 
WDRSD 2 
WDRSD 2 
WDRSD 2 
WDRSD 2 
DMP 2 
DMP 2 
DMP 2 
DMP 2 



90 
90, 

90 
66 
66 



60 

84 
84 

78 



1,00 (ll) c 
.73 (11) 

.74 (14) 



.47 (17) 



.88 (7) 



.99 (74) 
.99 (74) 

.98 (64) 



.80 (5) 
.71 (5) 

.50 (12) 



.74 (25) 1.00 (18) 



.97 (67) 



.25 (4) 



.93 
.81 

.74 



74 



,70 



,78 
,74 

,72 



,76 



.67 





Mean 






.76 




.V3 


.65 


*> 

.76 


.73 










Friday, 


March 17, 


1978 








10 


90 


DMP 5 


90 


.10 


(1) 


.91 (76) 


.R5 (13) 


.6? 


.57 


20 


90 


DMP 5 


90 


.00 


(1) 


.97 (76) 


.92 (13) 


.63 


.91 


30 


80 


DMP 5 


90 


.93 


(14) 


.99 (74) 


.00 (2) 


.64 


.81 


40 


80 


DMP 5 


90 


.93 


(14) 


.99 (75) 


1.00 (2) 


.97 


.89 


10 


90 


WDRSD 2 


66 


.71 


(14) 


.94 (44) 


• .89 <8) 


.85 


.84 


2C 


90 


WDRSD 2 


66 


.00 


(14) 


.76 (44) 


1.00 (8) 


.59 


.47 


30 


80 


•WDRSD 2 


60 














40 


80 


WDRSD 2 


60 












.49 * 


10 


90 


DMP 2 


84 


.00 


(1) 


.99 (76) 


1.00 (7) 


.66 


20 


90 


DMP 2 


84 


.00 


(1) 


.99 (76) 


.78 (7) 


.59 


.54 


30 


80 


DMP 2 


84 














40 


80 


DMP 2 


84 
















Mean 






.33 




.94 


.80 


.69 


.69 



b 

c 



Observers 10 and 20 observed at -WDRSD schools. 
Observers 30 and 40 observed at DMP schools. 

Observer did not attend first aay of the retraining session. 
Events coded by the expert. 



ERIC 
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TABLE 18 



Summary of the Category Marginal, Overall Marginal 
and Cohen's Kappa Observer Agreement Coefficients for 
the DMP and WDRSD Observers with the Expert Observers 

Retraining Session: March 16 and 17, 1978 
Other Materials Category 



Observer 
Pair 



Curriculum 
and 



Total 
Events 



Other 
Materials 



CateRory Marginal ; 



Other 
Materials 



Other 



Overall Cohen's 
Marginal Kappa K 



s a 


Expert 


Grade 


Observed 


Used 


Not Used 















Thursday, March 


16 1978 








i n 

JLU 


on 


nMP S 


90 


.92 (12) c 


.97 (73) 


.80 (5) 


.90 


.93 


on 
xu 




nMP S 


90 


.92 (12) 


.96*(73) 


.71 (5) 


.86 


.84 




ftn 


nMP S 














/.n 


pn 


nMP ^ 


90 












i n 
1U 


on 


LTTkT? Q n 0 


66 












On 
XU 


Qn 




66 












**n 


an 














• 


**u 


fln 

ou 


unw ^n 9 


60 












i n 

XU 


Qn 


PMP 2 


84 












on 
xu 


Qn 


nMP 2 


84 












in 

JU 


Rn 


nMP 0 














AO 


80 


DMP 2 


78 


1.00 (2) 


.96 (72) 


.25 (4) 


.74 


.65 




Mean 






.95 


.73 


.59 " 


.83 


.81 










Friday, March 


17, 1978 








10 


90 


DMP 5 


90 


.33 (6) 


.92 (71) 


.85 (13) 


.70 


.78 


20 


90 


DMP 5 


90 


.00 (6) 


.91, (71) 


.92 (13) 


.61 


.74 


30 


80 


DMP 5 


90 












AO 


80 


DMP 5 


90 












10 


90 


WDRSD 2 


66 


1.00 (1) 


.98 (57) 


.89 (8) 


- .96 


.82 


20 


90 


WDRSD 2 


66 


.00 (1) 


.98 (57) 


1.00 (8) 


.66 


.93 


30 


80 


WDRSD 2 


60 


4 










AO 


80 


WDRSD 2 


60 , 


.56 (1) 










10 


90 


DMP 2 


84 


.99 (76) 


1.00 (7) 


.83 


.55 


20 


90 


DMP 2 


84 


1.00 (1) 


.97 (76) 


.78 (7) 


.92 


.51, 


•30 


80 


DMP 2 


84 


.00 (1) 


.99 (80) 


1.00 (3) 


.66 


.85 


AO 


80 


DMP 2 


84 


1.00 (1) 


1.00 (80) 


l'.OO (3) 


1.00 


1.00 



Mea 



.48 



.97 



.93 



.79 



.77 



b 

c 



Observers 10 and 20 observed at WDRSD schools. 
Observers 30 and 40 observed at DMP schools. 

Observer did not attend first day of the retraining session. 
Events coded by the expert. 



9 
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TABLE 19 

Summary of the Category Marginal, Overall Marginal 
and Cohen's Kappa Onserver Agreement Coefficients for 
the DMP and WDRSD Observers with the Expert Observers 

Retraining Session: March 16 and 17, 1978 

Learner Moves 



lair 



70 
: j 
40 
ID 
20 
30 
4 0 
10 
20 
30 
40 



Curriculum 

;:a2 
•rt rt.*!.- 



Total 

EvetUs Written 



OraL 



CaU'^'ry Material 
Cover l UUi«r im 



Waiting 



Oil - 



No 

.t.\l I'Mvt 



Uvcral L i\»i:en s 
Marginal k*ip|>a n 



a 



9 

ERIC 



in 

9 j 
c) 
i'j 

90 

90 
bO 
^0 



i v 5 

2 

u:;.-;su 2 

L 2 

i,:::* 2 

D:\i> 2 



90 
6o 



60 
84 
84 



____ Thur.sd.iv* >Uivh 16, 1m /8 _ 

,90(2 l O c .7-5(8) . 7 r U 1 S ) .88(15) 
.90(29) 1.00(3) .88(15) .83(15) 



.94(15) .50(2) .90(30) 
1.00 (4) .67(6) .97(38) 
.75 (4) 1.00(6) 1.00(38) 



L.Uu(L0j 
.76125) 
1.00(25) 



•60(b) 
,80(5) 
.71(5) 



.94(170 
.78(21) 
.61(21) 



78 1.00(13) ,33(4) .63(18) 



.75 (J) 
.00 (0) 
.00 (0) 

.oU (J; 
1.00 t3) 
.75 (3) 

.83 (6) 



.92 



.67 



.86 



,61 



.31 ( >) 
1.00 (3) 


.88(1*)) 
.73(15) 


.80 (5) 
.71 (5) 


.76 
.87 


.53 
.57 


.68 (15) 
.00 (0) , 
.00 (0) 


.85(13) 
.83 (h) 
.86 (6) 


.50 (12) 
.67 (12) 
.83 (12) 


.73 
.59 
.63 


.65 
.59 
.68 


.TO (') 
.73 (ft) 
.89 (8) 


-o(0) 
.mU(1.i) 
.93(15) 


1.00 (18) 
.88 (7) 
.88 (7) 


.84 
.82 
.85 


./9 
.45 
.62 


.70 (7) 


.50 (6) 


.50 (4) 


.64 


.40 


.58 


.80 


.75 


.74 


.59 



1 J 


90 


D>tf 5 


90 


20 


50 




90 


30 


80 


ir.? 5 


90 


40 


80 


b:'.i j 5 


90 


10 


00 


U\.tt'J 2 


66 


20 


'jQ 


V.'il.-.jJ 2 


66 


30 


MJ 


«,.;;,..:> ^ 2 


60 


40 


Vi vy 


w..;;s"j 2 


60 


10 


90 


t.:*.i' 2 


84 


20 


90 


l:::' 2 


84 


jO 


b J 




84 


40 


80 


D. K u J 2 


64 



'r lda/. March 17. 1973 



.95(19) 

.63(19) 

.79(22) 

.96(22) 

.82 (9) 

.64 (9) 

.74(14) 

.88(14) 

.77(13) 

.54(L3) 

.94(16) 

. 70(16) 



1.00(1) 

1.00(1) 
.86(6) 

1.00(6) 
.38(3) 
.500) 
.67(2) 

1.00(2) 
.67(2) 
.33(2) 

1.00(2) 
.50(2) 



.96(22) 
.54(22) 
.81(31) 
.b4(31) 
1.00(25) 
.76(25) 
.52(23) 
.88(23) 
.80(24) 
.89(24) 
.87(34) 
.65(34) 



1.00 (8) 
- 1.U0 (8) 
.50 (2) 
,50 (2) 
.17 (6) 
.83 (6) 
.00 (0) 
.00 (0) 
.86 (6) 
.83 (6) 
.44 (9) 
.78 (9) 



.00 (0) 
.00 (Si) 
l.OO (1/) 
.59 117) 
.60 (5) 
.71 (5) 
.44 (8) 
.89 (8) 
.52 (25) 
.76 C/5) 
.67 (4) 
.K (4) 



.90(27) 
.96(27) 
.90(10) 
.80(10) 
.80(10) 
.70(10) 
.56 (9> 
.22 (9) 
.50 (7) 
.64 (7) 
.81(16) 
.89(16) 



.85 (13) 
.92 (13) 
.00 (2) 

1.00 (2) 
.80 (8) 

1.00 (8) 
.50 (4) 
.50 (4; 

1.00 (7) 
.78 (7) 

1.00 (3) 

1.00 (3) 



,81 
.74 
.69 
.81 
.65 
.74 
.49 
.62 
.73 
.68 
.82 
.78 



, on 
.63 
,54 
.59 
. 48 
.59 
.50 
.62 
.48 
.52 
.^6 
.61 



an 




Observers 10 and 20 observed at WDRSD schools. 
Observers 30 and AO observed at DMP schools. , 

Observer 30 did not attend first day of the 
retraining session. 



c Events coded by the expert. 

d No events were coded for this 
category by either the expert or 
jthe observer. 
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TABLE 20 



Summary of the Category Marginal, Overall Marginal 
and Cohen's Kappa Observer Agreement Coefficients for 
the DMP and WDRSD Observers with the Expert Observers 

Retraining Session! March 16 and 17, 1978 
Interactions 



Observer Curriculum 

Pair and 
Qbs a Exper t Grade 



10 
20 
30 
AO 
10 
20 
30 
AO 
10 
20 
30 
AO 



Total 
Events 
Observed 



Category Ma r ginal 



Target to 
Student 



Teacher to 
Group 



None of 
the above 



Overall Cohen's 
Marginal Kappa K. 



Thursday t March 16, 1978 



90 
90 
80 
80 
90 
90 
80 
80 
90 
90 
80 
80 



DMP 5 
DMP 5 
DMP 5 
DMP 5 
\7DRSD 2 
WDRSD 2 
WDRSD 2 
WDRSD 2 
DMP 2 . 
DMP 2 
DMP 2 
DMP 2 



90 

90 
_b 

90 
66 
66 

60 
8A 
84 

78 



.57 (7) c 
.50 (7) 

.50 (2) 
.50 (2) 
.40 (2) 

1.00 (1) 
.60 (5) 
.83 (5) 

.18 (2) 



.00 (0) 
.00 (0) 

.81 (13) 
.97 (28) 
.71 (28) 

.5/ (8) 
1.00 (2) 
.50 (2) 

.75 (9) 



.95 (79) 
.98 (79) 

.93 (70) 
.91 (32) 
.89 (32) 

.86 (43) 
.97 (73) 
.96 (73) 

.82 (61) 



.57 
.47 

.54 
.58 
.46 

.70 
.60 
.38 

.57 



b 

c 
d 



.40 
.46 

.5A 
.5A 
.40 

.58 
.46 
.54 

.3.0 





Mean 








.56 




.59 


.92 




>54 


.47 


Friday. March 17, 1978 


10 


90 


DMP 


5 


90 


.00 


(1) 


- d (0) 


.97 


(86) 


.19 


-.01 


20 


90 


DMP 


5 


90 


.00 


(1) 


- ' (0) 


.98 


(86) 


.30 


.32 


30 


80 


DMP 


5 


90 


.12 


(1) 


.75 (12) 


.92 


(72) 


. .38 


.36 


AO 


80 


DMP 


5 


90 


.25 


(1) 


.54 (12) 


.78 


(72) 


.50 


.36 


10 


90 


WDRSD 2 


66 


.43 


(3) 


.80 (12) 


.81 


(48) 


.38 


.42 


20 


90 


WDRSD 2 


66 


.50 


(3) 


.86 (12) 


.94 


(48) 


.38 


.54 


30 


80 


WDRSD 2 


60 


.50 


(2) 


.30 (10) 


.90 


(45) 


.74 


.45 


AO 


80 


WDRSI 


2 


60 


1.00 


(2) 


.90 (10) 


.98 


(45) 


.65 


.54 


10 


90 


DMP 


2 


8A 


.00 


(0) 


.80 (4) 


.99 


(75) 


. .47 


.49 


20 


90 


DMP 


2 


8A 


.00 


(0) 


.75 (4) 


.99 


(75) 


.42, 


.44 


30 


80 


DMP 


2 


84 


.50 


(2) 


1.00 (9) 


.97 


(68) 


.35 ' 


.32 


AO 


80 


DMH 


2 


8A 


1.00 


(2) 


1.00 (9) 


1.00 


(68) 


.54 


.42 


Mean 






.36 




. -77 


.94 




.44 


.39 



ERIC 



Observeis 10 and 20 observed at WDRSD schools. 
Observe/s 30 and 40 observed at DMP schools. 

Observer 30 did not attend first day of retraining session. 
Events coded by the expert. 

No events were coded for this category by either the expert or the observer, 
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TABLE 21 



57 



Summary of Means of the Category 
Marginal for Each Day of the WDRSD and DMP 
Training and Retraining Session 



Observation Category 



Training 
Thursday Friday 



R etraining 
Thursday Friday 



Self Paced 
Other Paced 
Individual 
Pair 

Small Group 
Large Group 
Paper/pencil 
Manipulatives 
Games 

Audio Visual 
Printed Material 
Other Material 
Engaged -Written 
Engaged -Oral 
Engaged-Covert 
Non-engaged Interim 
Non-engaged. Waiting 
Non-engaged Off -task 
Target to Student 
Teacher to Group 



.94 


.80 


.90 


.87 


.85 


.83 


.70 


.82 


.84 


.82 


.83 


.88 


.64 


.53 


.60 


.76 


.70 


.40 


.35 


.44 


.84 


.66 


.78 


.77 


.94 


.80 


.88 


.92 


.75 


.72 


.19 


.49 




.24 


.00 


.39 




.00 




.00 


.36 


.26 


' .76 


.33 






.95 ♦ 


.48 


.78 


.65 


.92 


.78 


.39 


.35 


.67 


.74 


.82 


.83 


.86 


.81 


.76 


.46 


.63 


.58 


.49 


.84 


.58 


.58 


.72 


.73 


.80 


.12 


.67 


.61 


.56 


.36 


.77 


..45 


.59 


.77 
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TABLE 22 



Summary of Means of the Cohen's Kappa 
for Sets of Categories By Day for the 
WDRSD and DMP Training and Retraining Sessions 



Observation Category 
Set 



Training 
Thursday Friday 



Retraining 
Thursday Friday 



Pac ing 


.76 


.66 


.65 


.76 


Grouping 


.74 


,64 


.67 


.74 


Paper /pencil 


.70 


.65 


.74 . 


.75 


Manipulatives 


.68 


.62 


.44 


.63 


Game 




.54 


.50 


.51 


Audio Visual 




.83 




.82 


Printed Material 


.59 


.57 


.73 


.69 


Other Material 






.81 


.77 


Learner Moves 


.51 


.52 


.59 


.56 


Interactions 


•51 


.38 


.47 


.39 
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TABLE 23 
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Summary of Generalizability Coefficients For 
Observer by Class by Day for the 
WDRSD and DMP Retraining Session 



Observation Category Observer Group 



10 x 20 x 90 40 x 80 



General Content 
Self Paced 
Other Paced 
Individual 
Pair 

Small Group 
Large Group 
Paper /pencil 
Engaged Written 
Engaged Oral 
Engaged Covert 
Total Engagement 
Non-engaged Inter im 
Non-engaged Waiting 
Non-engaged Off Task 
Total Non-engagement 



.00 


.96 


1.00 


1.00 


.99 


.99 


1.00 


.99 


.34 


.74 


.88 


.98 


.98 


1.00 


.99 


.96 


.94 


.81 


.12 


.00 


.95 


.60 


.46 


.86 


.98 


.63 


.96 


.99 


.94 


.76 


.98 


.96 
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On-Site Visits 

The summary of the agreement coefficients from the on-site visits 
to each of the four schools are given in Tables 24 through 27. Listed 
for each category where at least one event was coded by an observer is 
the marginal agreement. Then for the each set of categories, the 
category marginal agreements and Cohen's K are listed. The contingency 
tables showing the frequencies for each category for the on-site visits 
are in Appendix C. 

On-site visits were made to each school by one of the Phase IV 
staff members as soon as possible after the training session. The purpose 
of these visits was to check on the observer's facility in using the 
observation system and to discuss specific questions that may have arisen 
after the observer had done some observations at his or her school. This 
first on-site visit was considered as a part of the training procedures. 
A second on-site visit was made to three of the schools after the re- 
training session. This visit was made to check on the observer agreement 
after the observer was familiar with the system «nd had done several ob- 
servations. A second on-site visit was not trade at School 440 because 
observer AO had always shown a high degree of facility in using the 
system and it was difficult to arrange a time for one of the experts to 
visit the school. For each visit, observations were generally made at 
both Grades 2 and 5. Sometimes this was impossible and only one grade 
level was observed. The retraining session was during week 9. The 
first tests were administered during week 1 so that the first observations 

A 
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were done during week^Z~ — Thus on-site visits during weeks 2 through 8 
were done before the retraining sessions. Visits made during weeks 10 
through 16 were made^fter the retraining session. 



• \ 

rally_3 



School 452 (Observer 10) . For the on-site visit during week 5, 
the marginal agreements for General Content and pacing are high, genera 
above .80. The coefficients for the grouping are reasonably high except 
for pair and small group which both occurred very infrequently. The * K 
only material observed being used was paper/pencil "V.ch was coded with 
agreement for grade 5 (.96) but with lower agreement for grade 2 (.60). 
The high category coefficients for paper and pencil materials at grade 2 
indicate that there was high agreement when paper/pencil materials were 
not in use. There was some disagreement on the learner move categories 
indicating that the observer still had not gained adequate facility in 
distinguishing among the categories. There was relatively high agreement 
in coding when students were engaged (.95 and .74) and noa-engaged 
(.#7 and .73). For grade 2 the agreements for the teacher to group 
and for no interactions are high. However, for grade 5 these are much 
lower indicating the observer still lacked some facility in coding inter- 
actions consistently. 

The category Cohen f s k for grade 2 for week 5 are ^11 relatively 
high (above .73). This indicates that even though there was low agreement 
on some of the individual categories within a set (e.g. pair, small group, 
and engaged-oral) t^ese did not occur very frequently and that the observers 
agreed in coding many of the events. The Cohenfs K f s for grade 5 were 



70 



« 



71 



Categories 



General Content 
Self Paced 
Other Paced 
Individual 
Pair 

Small Group 
Large Group 
Paper /pencil 
Manipulatives 
Games 

Audio-Visual 
Printed Materials 
Other Materials 
Engaged-written 
Engaged-oral 
Engaged -covert 
Total Engagement 
Non-engaged interim 
Non-engaged waiting 
Non-engaged off -task 
Total Non-engagement 
Target to student 
Teacher to group 
No Interactions 



TABLE 24 

Observer Agreement Coefficients for 
On-Site Visits to WDRSD School 452 
Observers 10 and 90 



to 



Grade 2 



Grade 5 



Grade 2 



Week 5 Day 3 
(Total Events 72 ) 



• Week 5 ' Day 3 
(Total Events 66) 



Week ,15 Day 5 
(Total Events 72) 



Marginal Category Cohen's Marginal Category Cohen's 
Agreement' Agreemenc Kappa K Agreement Agreement Kappa K 



,.86 (18) 
.86 (6) 
.87 (31) 
1.00 (6) 



.50 

• CD 



(2) 
(1) 



.96 (28) 
.60 (10) 



.17 (6) 
.00 (0) 
.89 (16) 
.95 (22) 
.50 (3) 
.00 (5) 
1.00 (7) 
.87 (15) 
.00 (0) 
.86 (7) 
.98 (59) 



.80 
.88 

.68 



.83 



.50 



.44 



.90 
.83 

.84 



.86. 



.7: 



.73 



.75 (4) 

1.00 (14) 

.98 (4fr) 

.77 (10) 

.00 (0) 

.94 (52) 

.96 (47) 



.75 
.91 

.62 



.88 



.40 (10) .66 



.54 (19) 
.74^(29) 

1.00 (3) 

.48 (21) 

.82 (9) 

.73 (33) 

.00 (1) 

.38 (29) 

.67 (31 



.30 



.85 



.68 



.93 



.40 



.36 



Marginal Category Cohen's 
Agreement A greement Kappa K" 



1.00 (27) 

.88 (14) 

.94 (31) 

.62 (8) 

.90 (10) 

.50 (6) 

.95 (21) 

.96 (24) 



.00 (2) 

.88 (8) 

1.00 (3) 

.96 (24) 

.94 (35) 

1.00 (2J 



.40 
.38 



(5) 
(3) 



.83 (10) 

.00 (0) 

.88 (14) 

.94 (52) 



.89 
.94 

.79 



.97 

.65 
.80 



.38 



.98 
.91 

.77 



.98 



.94 
.77 



.61 
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a Events coded by the expert. 
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generally lower, particularly for pacing (.56), grouping (.68), learner 
moves (.40) and for interactions (.36), Thus the observer at this point 
had not yet gained the expected facility JLnusfhg the system consistently. 
The coding of these two classes were discussed in detail 4 urin g the * 
on-site visit. 

The agreement coefficients were generally higher in coding grade 2 
during week 15. Only .individual, small group, non-engaged waiting and 
non-engaged off -task have low marginal coefficients. The Cohen's K f s for 
four of the sets of categories are extremely high (above .90) and for the 
other three sets- of categories are moderately high (above .60). Thus the 
ability of .Observer J.0 had improved after using the system and had reached 
aiv-Acceii table level in coding most of the categories. 

School 504 (Observer 20 ). The marginal agreement coefficients for 
Observer 20 (Table 25) were high except for the learner move categories. 
Observer 20 had difficulty in distinguishing between engaged-^yert and 
non-eiigaged waiting. This resulted in a low agreement coefficient for 
each of the categories and for total engagement and total non-engagement. 
Ihe Cohen's K also reflect a high facility in coding all the sets of 
categories except for learner moves (k - .47). 

Observer 20 had maintained her ability in coding the categories other 
than learner moves and also had improved "greatly in coding learner moves 

by the time of the on-site visit after the retraining session, week 15. 

**** 

The marginal agreement coefficients, with the exception of a few with low * 
frequencies, are high for both grade levels. The Cohen's K ! s are all high 
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Categories 



eitt 



General Conte 
Self Paced ' 
Other Paced 
Individual . 
Pair 

Small Group 
Large Group. 
Paper/pencil 
Manipulatives . 

Games > 
Audio-Visual « 
Printed Materials 
Other Materials 
Engaged-written 
Engaged-oral 
Engaged-c overt 
Total Engagement 
Non-engaged interim 
Non-engaged waiting 
Non-engaged off -task 
Total Non-engagement 
Target to student 
Teacher to group 
No Interactions 



1\ 



TABLE 25 

Observer Agreement Coefficients for 
On-Site Visits to WDRSD School 504 
Observers 20 and 90 



Grade 2 
Week 4 Day 1 
(Total Eve nts 54) 



Grade 2 
Week 15 Day 2" 
(Total Events 54) 



Marginal Category Cohen's 
Agreement Agreement Kappa K 



Marginal. Category Cohen's 
Agreement Agreement Kappa K 



.88 (8) a 
.74 (14) 
.88 (32) 
.89 (16) 
.00 (0) 

.93 (30) 
.94 (29) 



.80 (8) 
1.00 (1) 
.59 (13) 
-.67 (/27 
.50 XIV 
.50 (22) 
1.00 (1) 
.58 (24) 
.00 (0) 
.80 (8) 
.96 (45) 



.74 
.83 

.67 



.92 



75- 



.74 



.93 
.81 

.81 



.90 



.47 



.93 



1.00 (10) 
.92 (11) 
.97 (33) 
.90 (9) 

1.00 
.00 (1) 

1.00 (32) 
•1.00 (12) 



1.00 
.96 

.78 



1.00 



1.00 
1.00 
.67 
.50 
1.00 
1.00 
.57 
, .80 
.88 
1.00 
1.00 
.87 
.97 



(4) 
(4) 
(2) 
(2) 
(18) 
(22) 
(7) 
(8) 
(7) 
(22) 

(1) 
(13) 
(39) 



1.00 
* .97 

.90 



1.00 



1.00 
1.00 
.77 



1.00 
1.00 
.70 
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.65 



Grade 5 
Week 15 » Day 2 
( Total Events 138) 



Marginal Category Cohen's 
Agreement Agreement Kappa < 



1.00 (22) 
.95 (88) 
.00 (5) 
.-99 ,(82) 
,91 (10) 

1.00 (1) 



1.00 
.65 

.97 



.89 (42) .93 



.87 (71) 
.00 (1) 
.68 (15) 
.00 (0) 
.85 (52) 
.97 (67) 
.91 (10) 

.81 (16) 
.92 (26) 
.00 (0) 

1.00 (135) 



..86 
.66 
.71 



.20 



T 



1.00 
.92 

:96 



.84 



.85 
.98 
.84 



.33 



Events coded by the expert. 
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(above .70) except for interactions where there was some disagreement as 
to what the interaction was on a very small number of events. There was 

extremely high agreement when there were no interactions. Thus Observer 20 
had gained a very high level of facility in using the system. 

School 433 (Observer 30 ). Observer 30 was having more difficulty 
with th6 observation system (Table 26) by^ week 4 than did the other three 
observers. There is some question about the meaning of these coefficients 
for this on-site visit since early in tlie morning the observer's apartment 
was destroyed by fire which caused some distraction to the observer. The 

observer did have a misconception of individual grquping. Even though 

\ 

students were working by themselves on worksheets, the observer coded this 
as a large group activity since all the students were doing the same work- 
sheet, fhis misinterpretation of grouping was discussed during the visit. 
By week J.6 the observer had gained a high level of facility in using the 
system as indicated by t;he high marginal coefficients for most categories. 
Those categories with low coefficients were only those coded very infre- 
quently by the expert observer. The Cohen's K for the materials categories 
of games, printed materials and other do not have much meaning since these 
categories were observed so infrequently. 

School 440 (Observer 40) . With, the exception of learner moves, the 
agreement coefficients for Observer 40 are extremely high (Table 27) for 
the on-site visit during wesk 4. Observer 40 had some difficulty in dis- 
tinguishing among engagel-covert and the other learner moves. She was not 

i 
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TABLE 26 
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Observer Agreement Coefficients for 
On-site Visits to DMP School 433 
Observers 30 and 30 



Categories 



Grade 2, ^ 

Week 4 Day 4 
(Total Events 54) 



Marginal Category Cohen 1 s 
Agreement Agreement Kappa K 



Grade 5 

Week 16 Day 2 

(Total Events 96) 

Marginal Category Cohen 1 s 
Agreement Agreement Kappa K 



General Content 
Self Paced 
Other Paced 
Individual 
Pair 

Small Group 
Large Group 
Paper /pencil 
Manipulatives 
Games 

Audio Visual 
Printed Materials 
Other Materials 
Engaged-Written 
Engaged-Oral 
Engaged -Covert 
Total Engagement 
Non-engaged Interim 
Non-engaged Waiting 
Non-engaged Off Task 
Total Non-engagement 
Target to Student 
Teacher to Group 
No Interactions 



.67 (2) a 


.82 - 


.79 


.60 (5) 


.37 


.48 


.04 (1) 


.42 


.15 


1.00 (21) 


.86 


.87 


.55 (51) 






.97 (70) 




.88 


.04 (1) 


.42 


.15 


.90 (19) 


.80 


- (0) 






.50 (2) 






- (0) 






1.00 (2) 




• 


.55 (51) 






.99 (68) 




.90 


.98 (52) 


.82 


.79 


1.00 (41) 


.85 








.67 (2) 


.75 


.68 








.00 


.52 


.42 



.94 (17) 
.50 (2) 
.57 (21) 
.78 (40) 
.67 (3) 
.61 (8) 
.20 (1) 
.60 (12) 

.58 (12) 
.95 (40) 



.60 



.65 



.54 



.46 



.00 (1) 
.75 (12) 
.25 (1) 
.94 (47) 
.94 (60) 
1.00 (5) 
1.00 (18) 
.75 (8) 
.94 (31) 
.40 (2) 
.82 (27) 
.97 (66) 



.53 
.76 



.44 



.42 
.65 



.60 



a Events coded by the expert. 

b No events cod.ed by either the observer or the expert. 
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Observer Agreement Coefficients on Observation Categories 
for On-Site Visit for DMP School 440 
Observers 40 and 80 



Categories 



Grade 5 

Week 4 Day 4 
(Total Events 96) 



Marginal 
Agreement 



Category 
Marginal 



Cohen's 
Kappa k 



General Content 
Self Paced 
Other Paced 
Individual 
Pair 

Small Group / 
Large Group j 
Paper /Pencil 
Manipulatives 
Games , 
Audio Visual 
Printed Materials 
Other Materials 
Engaged -Written 
Engaged -Oral 
Engaged -Covert 
Total Engagement 
Non-engaged Interim 
Non-engaged Waiting 
Non-engaged Off-task 
Total Non-engagement 
Target to Student 
Teacher to Group 
No interactions 



1. 00 (9) 
\99.(77) 
.9,1 (10) 
.99 (77) 



.91 (10) \ 
.95 (73) 



.94 (17) 
1.00 (1) 
.79 (34) 
.00 (0) 
.38 (26) 
.92 (60) 
.50 (2) 
.46 (6) 
.79 (19) 
.84 (27) 
.00 (2) 

.96 (84) 



1.00 
.96 

.96 



.89 



.98 
1.00 
.56 



1.00 
.97 

.97 



.77 



.98 
1.00 
.66 



.49 



.63 



a Events coded by the expert. 
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» as specific on whether or not the student was actually writing or only 
was reading the problem with a pencil in hand. This caused some dis- 
agreement on these specific categories, however agreement on total 
engagement (.92) and total non-engagement (.84) was high. The Cohen's k's 
for the sets of categories were extremely high for general content (1.00), 
pacing (.97), grouping (.97), printed materials (.98) and other materials 
(1.00). The Cohen's k's were moderately high for the other categories - 
paper/pencil (.77), learner moves (.66), and interactions (.63). 

On-site visits Summary . The results of the on-site visits 

indicate that the observers still had difficulty in coding some categories 

i 

after the training session. The coding of learner moves seems to require 
more time than was given in the training session, for observers to gain 
facility in distinguishing among the categories.' The observers could, . 
however, generally distinguish whether or not siudents were engaged. All 
of the observers had reached a high degree of facility in using the / 
system by th- time of the second period, after, the retraining session. 
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VI 

' * Conclusion s 

The observation categories that had high interobserver agreements 
for the training session, retraining sessions, and on-site visits are: 

Self -paced 

Other-paced 

Individual 

Large Group 

Paper/pencil 

Engaged-written 

Engaged-covert 

Non-engaged off-task 

For the other categories where there were lower interobserver 
agreements, most of these categories occurred very infrequently. Inter- 
observer agreement was relatively high in coding pairs and small groups 
of students that were formally designated by the teacher. Disagreement 
occurred when there were occasional pairs and small groups formed to 
have a question answered and only existed for a short period of time. 
Thus pair and small group time should be iaferpr^ted as time spent in 
these groupings as directed by the teacher and for a measurable amount 
of Lime of the class period. 

Material categories other than paper/pencil occurred very infre- 
quently. There. was high agreement that these materials were not used, 
but the lower agreement on when these materials were used reflects the 



00 




low frequencies of occurrence of the different types of materials in 
the classes observed. The most important agreement on the material 
category is the high agreement that the materials were not used. 

Three learner moves (engaged-oral, non-engaged interim and non- 
engaged waiting) had low interobserver agreements. Students were 
rarely engaged in talking. Thus there were very few instances of 
engaged-oral for the observers to reach agreement. However, there was 
improvement on this category from the training session to the retraining 
session. Non-engaged interim is a, more nebulous ; category which the 
low interobserver agreements- on this category reflect. Non-engaged 
waiting also appears to be less well defined as reflected by the low 
interobserver agreements. However, when the three engaged categories 
and the three non-engaged cagegories are combined into two groups, 
total engagement and total non-engagement, the agreement is generally 
high as reflected by the generalizability coefficients for the retraining 
session. 
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APPENDIX A 



Contingency Tables for each set of Observation Categories, 
for Pairs of ' Observers for the WDRSD and DMP Training Session 

(January 10-13 1978) 



Appendix A 
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Tables 
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Printed Materials 
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Learner Moves 
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Phase IV - IOE Evaluation Project 
Descriptive stu^y - spring 19T8 

Rellebllfty cm oenerai Content 
Observer 10 Expert 90 
School box orede a 
Week 0 Dey * 

WORSO 



Cetegory chosen by observer 10 



Cettgory chosen 
by Expert 90 


cat. 

M 


1 


2 


3 




s 


6 


, T 


Merelnel 
Toteis 








> + + + + + + 


Malt 


1 i 


► 0 


0 


0 


t 

0 


0 


^0 
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► 9 


Trana 1 1 ion 
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0 


0 


0 


0 


• 0 < 


► 0 
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► 0 
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0 


0 


0 


0 


0 < 


► 0 


Break 
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> o, 
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0 


0 
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0 


0 i 


► , o 
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0 
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0 
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0 
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► 0 


Other Academic 
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► 0 
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0 


0 


0 


0 


0 


► 0 
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* < 


7 


» 0' 


0 

4 


0 


t 

0 


* 


0 




► 5* 


Marginal 


Toteis < 

1 1 


t 0 


0 


0 


0 


0 


0 


3* 


> # 54 



LloKt'i 

Nominal Marginal • CMtanilon 
Agraamant Agraamant ©# KaPPa 



1.000 



1.000 



Catagory Marginal Aoraamantl 1.000 
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Table 29 

Phots IV - IOC Evaluation ProjocV 
DaicrlPtlva Study - Spring 1IT6 

pal lability on Oanaral Contant 
Obtarvar SO Export 90 
school aoi Orada I 
WaaK 0 Day * 

W0R30 



oc 



Catagory chooan Cat# 
by EMPart 90 * 



Cataoory choton by Obtarvar 20 
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Marginal 
Totalt 



♦ • 
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CKtant Ion 
of Kappa 



Halt 


\ 
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0 


♦ 
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0 
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1 


1 


0 
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♦ 
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Table 30 

* 

Phaia IV • ige Evaluation Projact 
OatcriPtlva study - soring 1*76 

pal lability on oanaral contant 
Obtarvar 30 Expart 60 
School 601 Orada 3 
Waak 0 Oay 4 

WORSO 



Cataoory ehosan Cat* 
by Expbrt 60 # 



Catagory chosan by obtarvar 30 

1 2 3 * 5 6 ,7 



Halt 


1 


♦ 2 


0 


0 


0 


0 

i 


0 


1 


Tram \ t ion 


2 


♦ 0 




0 


0 


0 


0 


i 


Panagamtant 


3 


♦ 0 


0 


5 1 


0 


0 


0 


1 


■ raak 




♦ 0 


0 


0 


0 


0 


0 


0 


Monacadomic 


5 


♦ 0 


0 


0 


0 


p 


0 


0 


Othar Acadamic 


6 


♦ 0 


0 


0 


0 


0 


0 


0 


fcoita ol tho Abova 


7 


♦ 0 


0 


0 


1 . 


0 


0 


*s 






♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦ ♦♦♦♦♦♦♦ 


Marginal 


Totals 


♦ 2 


* 


1 


1 


0 


0 


62 



♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 



Marginal 
Totals 

♦♦♦♦♦♦♦♦ 

3 

6 

2 

0 

0 

0 

49 

♦♦♦♦♦♦♦♦ 
60 



LlftM'l 

Nominal Marginal Extanslon 
Agraamant Agraamant of K«PPa 



• 667 

• 667 

• 500 



• 667 

• 667 

0 

,600 
0*000 



,655 
,6*3 
,492 



.960 



Cataaory Marginal AOraamanti »65i 
Cohan Kappa I »706 
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Table 31 



oo 
o 



Phata IV - !0E Evaluation Projact 
OaicMPtlva Study * Soring 1978 

Reliability on Oanaral Contant 
Obiarvar to Expart SO 
School SOI orada t 
waak o Day *> 

WDRSD 



Cataoorv chotan by Obtarvar *>0 



LIOHtU 



Category ehoten 
by export to 


Cat* 
# 


1 


8 


3 


a 


s 


« 


T 


Marginal 
Totals 


Nominal 
Agreement 


Marginal 
Agreement 


Extent lo 
ol KapPa 
















MOlt 


1 < 


► 8 


0 


0 


0 


0 

A 


0 


1 < 


► 3 ♦ 


*66T 


• OAT 


,688 


T^oniit Ion 


2 < 


► 0 


« 


0 


0 


0 


0 


0 < 


► J* ♦ 

► / ♦ 

► / a ♦ 

► \ o ♦ 

► V * 


1*000 


1*0*00 


1,000 


Mnegement 
■rook 

Nonoeodomic 
Other Academic 


3 • 


► 0 


0 


0 


0 


0 


0 


z ■ * 


0*000 


0*000 


0,000 


4 < 


► 0 


0 


0 


0 


0 


0 


0 < 




0*000 




8 < 


► 0 


0 


0 


0 


0 


. 0 


0 < 








« • 


► 0 


0 


0 


0 


0 


0 


0 < 


► 0 ♦ 








None ol the Above 


T < 


► 0 


0 


0 


1 


0 


0 


*8 < 


► *« ♦ 


• MO 


• 001 


,066 
















Merginel 


Total* < 


► 8 


6 


0 


1 


0 


0 


81 < 


- 60 ♦ 












►>♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦,♦♦»♦♦♦♦♦♦♦♦♦< 











Category Marginal Agreement I ,838 
Cohen it KaPpa I ,TT3 
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Table 32 

PHaM IV • IOE Evaluation Projact 
OateriPtlva s'tudy - spring 1976 

ft* I lability on Oanarai Contant 
Obiarvar 60 Export 80 
School 801 Orada I 
waak o Day * 

OMP 



Catagory ehotan 
by Export 80 



JK«I t 

Transition 
Managatoant 
Braak 

Nonacadamic 
Othar Acadamlc 
Nona o# tha Abova 



Cat* 
# 



Catagory ehoian by Obiarvar 60 
1 S 3 6 9 6 7 



Marginal 
Totals 



♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 



♦ 

1 ♦ 
♦ 

z ♦ 
♦ 

3 ♦ 
♦ 

6 ♦ 
♦ 

5 ♦ 
♦ 

6 ♦ 
♦ 

7 ♦ 
♦ 



3 

1 

1 

0 

0 

0_ 

0 



2 
S 

0 
0 
0 
0 
0 



0 

1 

3 

0 
0 
0 
0 



0 
0 
0 
0 
0 
0 
3 



0 
0 
0 
0 
0 
0 
0 



0 
0 
0 
0 
0 
0 
0 



0 
2 

1 

0 
0 
0 
63 



♦ 

♦ 
♦ 
♦ 
♦ 
♦ 
♦ 
♦ 
♦ 
♦ 
♦ 
♦ 
♦ 
♦ 



5 
9 
8 

0 
0 
0 
65 



i. ♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 



Marginal Total! ♦ 
♦ 



66 



♦ 
♦ 



86 



Nominal 



Marginal 



UQHt'i 
Extension 



Agraamant Agroomont of KaPPa 



#600 


1*000 


,575 


#556 


.778 


• 515 


• 600 


• BOO 


,560 




0*000 





,960 



.908 



,686 



Catagory Marginal Aaraamanti ,713 
COhanft Kappa I ,666 



•r»*r.ie 
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Table 33 



Phase IV - IOE Evaluation. project 
OeicrlPtlve Study - soring 19TS 

Reliability on oanaral Contont 
ObiarVar 20 • Export SO 
Schopi SOI Orado I 
w#ak o oay ♦ 
0MP 



Category ehoaon by Obiarvar 20 



Category chosen 
by Expert SO 



Halt 

Transition 
Nanagamant 
Sraak 

Nonocodonie 
Other Acedemle 
Aone of tha Above 



Cat, 
a 



♦ 

1 ♦ 
♦ 

2 ♦ 
♦ 

3 ♦ 
♦ 

a ♦ 
♦ 

s ♦ 
♦ 

6 ♦ 
♦ 

7 ♦ 
♦ 



Marginal Toton ♦ 



3 
0 
0 
0 
0 
0 
0 



1 

6 
1 

0 
0 
0 
0 



1 

0 

a 

0 
0 
0 
0 



0 
0 
0 
0 
0 
0 

1 



0 
0 
0 
0 
0 
0 
0 



0 
0 
0 
0 
0 
0 
0 



0 
3 
0 
0 
0 
0 
64 



67 



Light'! 

Marginal No" I no | Mors I no I Extension 
Toton Agreement Agreement of kopPo 



♦ 
♦ 
♦ 

♦ 
♦ 
♦ 
♦ 

♦ 
♦ 



8 
9 
S 

0 
0 
0 
68 



6* 



♦ 

♦ ,600 
♦ 

♦ ,667 
♦ 

♦ ,600 
♦ 

♦ "«••••• 
♦ 

« ...... 

♦ 

♦ ----- 

♦ 

♦ ,tSB 

♦ 



♦ 
♦ 



,600 
,669 

1,000 
0,000 



.585 
,632 
,767 



,970 



.98* 



Cataoory Marginal Agreement I ,692 
Cohen i • Kappa I ,773 
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Table 34 



Rhaie IV - JOE Evaluation project 
DetcrlPtlve study - spring 1978 

Reliability on General Content 
Observer 30 Expert 90 
School 801 Grade 8 
week o Dey * 
OMP 



Category choien by Observer 80 



Category choien Cat. 
by Expert 90 * 



Nal t 1 

Transition 2 

Management 3 

•reek * 

Nonacedomic 9 

Other Academic 6 

None of the Above T 

Marginal Total! 



Mar g I na I Norn I na | Merg I na I 



1 


2 1 


3 


* 


s 


6 


7 




T0t«|i 


Agreement 


Agreement 


♦ 0 


0 


0 


0 


0 


0 


0 


♦ 
♦ 


0 < 




0*000 


♦ 0 


' 0 


0 


0 


0 


0 


1 


♦ 
♦ 


I < 


► o.ooo 


0*000 


♦ 1 


0 


0 


0 


0 


0 


0 


♦ 
♦ 
♦ 


I '< 


► 0.000 


0.000 


♦ 0 


0 


0 


0 


0 


0 


0 


♦ 


0 




0.000 


♦ 0 


0 


0 


0 


0 


0 


3 


♦ 
♦ 


3 


» 0.000 • 


0,000 


♦ 0 


0 


0 


2 


0 


0 


0 


♦ 
♦ 


2 


» 0.000 


0*000 


♦ * 


0 


0 


0 


0 


0 


6T 


♦ 
♦ 


Tl 


» *9*k 


1.000 
















♦ 
















— \ 















LlOnt't 
Extent Ion 
Of KapPa 



8 o o a J V 0 71 ♦ ?• 
♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 

4 

category Marginal Agreement, .1*3 
Cohen »• Kappa I a 77 



DECK 



0.000 
0*000 

0.000 
0.000 
,371 



00 

-fr- 



iable 35 

Phata IV • IOC Evaluation Projact 
• OatcrlPtlva Study - Sprino 1976 

Ral lability on oanarai Contant 
Obtarvar %0 Expart 90 
School 801 Orada B 
Maak o Day * 
DMP 



Catagory chotan Cat, 
by Export 90 # 



Catagory chotan by Obtarvar 40 
1 9 3 * S ft 7 
♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 



Marginal 
7ota|i 



Lights 

Horn | na | Mar o I na I Extant I on 
Agroamant Agraamant of KaPPa 







♦ 
















♦ 


0 ♦ 








tacit 


1 


♦ 


0 


0 


0 


0 


0 


0 


0 


♦ 
♦ 




•— (r 




Transition 


2 


♦ 
♦ 


0 


1 


0 


0 


0 


0 


0 . 


♦ 
♦ 


i ♦ 


1.000 


1.000 


1.000 


Manaoamant 


3 


♦ 
♦ 
♦ 


0 


0 


1 


0 


0 


0 


0 


♦ 
♦ 


i ♦ 


1.000 


1.000 
0.000 


1,000 


■raax 


4 


♦ 


0 


0 


0 


0 


0 


0 


0 


♦ 
♦ 


0 ♦ 






Nonacadami c 


S 


♦ 
♦ 


0 


0 


0 


3 


0 


0 


0 


♦ 
♦ 


3 ♦ 


0.000 


0.000 


0.000 


Othar Acadamlc 


6 


♦ 
♦ 


0 


0 


0 


a 


0 


0 


0 


♦ 
♦ 


8 ♦ 


0.000 


0.000 


0.000 


Nona of tho Abovo 


7 


♦ 
♦ 


0 


0 


0 


0 


0 


0 


Tl 


♦ 

* 


Tl ♦ 


1.000 


1.000 


1.000 






A A 








Marginal 


Total! 


♦ 
♦ 


0 


1 


1 


5 


0 


0 


Tl 


♦ 
♦ 


Te ♦ 









catagory Marginal Agraamanti «S00 
Cohan ft Kappal 
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Table 36* 

phasa IV • XGE Evaluation Projact 
DascrlPtlvo Study • SPrino 1978 

Reliability on Oanaral Contant 
Obiarvar 10 Export 90 
School 601 Oroda S 
Waak o nay s 

DHP 



Catagorv chosan 
by EMpart 90 



Malt 

Transition 
Managomant 
Braaw 

Nonacadamic 
OtKar Acadamlc 
Nona o# tha Abova 



Ca 



Marginal Totals 



Cotagory choaan by Obaarvar 10 
1 2 3 4 5 6 T 



1 


0 


0 


0 


0 


0 


0 


0 


1 


1 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0, 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 


1 


0 


0 


0 


26 



26 



Marginal 
Totals 



♦ 
♦ 
♦ 
♦ 

♦ 

♦ 
♦ 
♦ 
♦ 
♦ 

♦ 
♦ 
♦ 



1 
2 

0 
0 
0 
0 
27 

30 



tlQHt'S 

Now I no I Maro I na I EMtans I on 
Agraamant Aoraamant 0# Kappa 



ItOOO 
• 500 



1*000 

•uoo 

0t000 



1,000 
9 4B3 



• 963 



9 t63 



• T2« 



Cataoory Marginal Aoraamanti #616 
eohants koppoi *69i 



BEST r 



Oo 



95 



\ 



Table 37 

Phaia IV - !0E evaluation Project 
OascriPtlva Study • spring. 19TB 



p«l lab l I | tv on 
Obiarver 20 
School 801 
WaaK 0 



Ganaral Cpntant 
EKPtrt 90 
Orada 8 
Day 5 

DMP 



L 



oo 



Catagory choian Cat* 
by CKoart 90 # 



Cotagory chotan by Obtarvar So 

1 ( 2 3 * 5 4 7 

♦♦♦♦♦♦^♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦4 
♦ 



Marginal 
Total a 

♦♦♦♦♦♦♦♦ 



Walt 


1 


♦ 


0 


0 


0 


0 


0 


0 


1 i 


► I 


Transition 


2 


♦ 
♦ 


0 


2 


a 


0 


0 


0 


0 < 


► s 


Managamant 


3 


♦ 
♦ 


0 


0 


0 


0 


0 




0 < 


► 0 


■raak 


4 


♦ 
♦ 


0 


0 


0 


0 


0 


0 


0 < 


► 0 


Kon«jcadtm|c 


5 


♦ 
♦ 


0 


0 


0 


0 


0 


0 


0 < 


► 0 


Othar Acadamlc 


ft 


♦ 
♦ 


0 


0 


0 


0 


0 


0 


0 i 


► 0 


Mont e# tha Abova 


7 


♦ 
♦ 


0 


0 


0 


0 


0 


0 


21 < 


► 27 



♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 
♦ 

Marginal Totals ♦ 020000 as 
♦ 

♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 



30 

♦♦♦♦♦♦♦♦ 



Light's 

Nominal Marginal CMtantlon 
Agraamant Agraawant of h^ppb 



OtOOO 
1*000 



0*000 
1*000 



0*000 
1.000 



i#ooo 



>9M 



1.000 



Cftaoory Marginal Agraamantl «45B 
Cohants Kappa I 
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Phaoe IV^ 0<>E Evaluation Project 
Descriptive Study • Spring 1*T6 



Reliability on General Content 
Observer 30 Expert BO 
School 601 Oradi 3 
week o Day 5 
OMP 



/ 



Category choten 
by, expert 00 


Cat* 
* 


1 


wait 


1 ♦ 


1 


Trent 1 1 1 on 
Management 


2 ♦ 

v. 


0 
0 


Greek 

honecadem ic * 


4 ♦ 

5 ♦ 


0 


«Other Academic 


e 6 ♦ 

V- ♦ 
* 7 ♦ 


0 


None of the Above 


0 



MeVQinal Toteit 



Category choten by Obterver 30 
2 3 * 5 6 T 



t^rglnal 
Totait 




0 
0 
1 
0 
0 
0 
0 



0 
0 
0 
0 
0 
0 
0 



0 

°/ 

0 
0 

, 0 
0 



0 
0 
0 
0 
0 
0 



0 
0 
0 
0 
0 
0 
2* 



♦ 
♦ 
♦ 
♦ 
♦ 
♦ 

♦ 
♦ 
♦ 
♦ 
♦ 
♦ 



i 

0 

1 

0 

' 0 
0 

a? 



♦ ♦♦ 




no I na i 
•Agreement 



iSOO 



liOOO 



>8fl9 



Light's 
Marginal Extension 
Agreement of KepPe 

. f soo «« fl3 
0*000 -hi— — 
loOOO ifOOO 

i 

OtOOO 



! 



Ceteoory Marginei Aoreewenti ♦ 
Cohen's Kappei ♦ Si* 



00 
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Table 39 

Phase IV - XOE Evaluation Project 
Descriptive study - Soring 1978 

Reliability on General Content 
Observer 40 Expert BO 
School 601 Grade 9 
Week o Day 3 
DMP 



oo 

00 



Category chosen Catt 
by Expert 00 * 



Category chosen by Observer, *o 
1 2 a . * 3 6 T 



Marginal 
Totals 



visit 


1 


♦ 


1 


0 


0 


0 


0 


0 


1 


Trent it ion 


2 


♦ 


0 


0 


0 


0 


0 


' 0 


0 


Management 


3 


♦ 

♦ 


0 


0 


1 


0 


0 


0 


0 


Break 


A' 


♦ 


0 


0 


0 


0 


0 


0 


0 




♦ 
















Nonecademjc 


5 


♦ ' 
* 


0 


0 


0 


0 


0 


0 


0 


Other Academic 


6 




0 


0 


0 


0 


0 


0 


0 


None of the Above 


7 


♦ 
♦ 


0 


0 


0 


0 


0 


0 


27 


*. 




+++++++++ 














1 Merginel 




















Toteis 




1 


0 


I 


0 


0 


0 


20 



4 ♦ 



♦ 
♦ 
♦ 

♦ 
♦ 
♦ 

♦ 
♦ 
♦ 
♦ 

♦ 

♦ 

♦ 



2 
0 
1 

0 
0 
27 

30 



Nomjnei 
Agreement 



,»«00 



1*000 



ltOOO 



Heroine) 
'Agreement 



,800 



1,000 



vUflht'l 
fx ten at on 
0# KOPPe 



,483 



1,000, 



i.ooo 



category Merginel a^reementi «•>! 
Cohen Keppal ,TIT 
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Table 40 

I k 

Phai* XV - IGE Evaluation projact 
OaicrlPtlva Study •> Spring 1976 

Rat lability on Oanaral Content 
Obiarvar 10 Expart 90 
School aoi Qrada 2 • 
waak 0 Day 5 

WDRSD 



Cftagory cHotan 
by Expert 90 



halt 

Tram 1 1 ion 
Managamant 
Braak * 
Nonecadamic 

9 

Ottvar Acadamic 
Nona o# tha Abova 



Cat, 



Cataoory choian by obiarvar 10 
1 2 3 4 5 6 T 



Mar a I na I 
Total • 



1 ♦ 

2 ♦ 
♦ 

3 ♦ 

4 ♦ 
♦ 

5 ♦ 
♦ 

6 ♦ 
♦ 

7 ♦ 
♦ 



0 
0 
0 
0 
0 
0 
0 



0 

J 

0 
0 
0 



Haroinal Totaii ♦ 0 
♦ 

♦♦♦♦♦♦♦♦♦ 



0 
0 
0 

•0 
0 
0 
0 



0 
0 
0 
0 
0 
0 
0 



0 

0 
0 
0 
0 
0 
0 



0 
0 
0 
0 
0 
0 
0 



0 

Hf 

0 
0 
, 0 
52 

52 



♦ 

♦ 
♦ 

♦ 
♦ 

♦ 
♦ 
♦ 

♦ 

♦ 
♦ 



c 
1 

0 
0 
0 
0 
53 

54 



Nominal 
Agraamant 



Marginal 
Agrae<nan{ 



LigM«t 
Extanilon 
el KaPPa 



1 ♦000, 



• 500 



1,000 



,951 



• 951 



t 49i 



f, 



Category, Marginal Agraamant I 
Cohann Kappas «•!)• 
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Table 41 

Phase IV - !0E evaluation Project 
DescrJPtlve Study • Soring 1978 

r Ral lability on General MRontent 
Observer 20 Expert 90 
School 601 oradg 2 
Week 0 Day 8 

^ WORSD 



Category chosen 
by Expert 90 



Cat.. 
* 



Melt 

Trent Itlon 
Management 
area* 

Nonacademie 
Otfcer Academic 
hone o$ the Above 



Category choeen by Observer 20 



♦ 

1 ♦ < 
♦ 

♦ 

♦ 

5 ♦ 

♦ 

6 ♦ 
♦ 

7 ♦ 
♦ 

♦ ♦♦♦♦ 
♦ 

werginel Toteis ♦ o 



o 

0 



Marginal 
Tote i • 




♦ 

• ♦ 
♦ 

♦ 
♦ 
♦ 
♦ 
♦ 
♦ 
♦ 
♦ 




♦ 4 X 



Nomine I Meroinel 
Agreement Agreement 



Light's 
Extension 
of KaPPe 



0*000 



1*000 



,9il 



1,000 



ii019 



category Marglnel Agreement! 1*000 
Cohen»e KaPpei^j^jt9' 



P 



9 
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Table 42 ' 

Phase IV - IOE Evaluation project 
Descriptive Study - Spring 197B- ' 

• Reliability on oenaral content 
Observer 30 Expert BO 
School 601 Orade I 
\ week o Day S 

WDRSD 



J . 



\ 



fcotegory chosen by obterver 30 



Category chosen 
by Expert 60 



tea 1 1 

Transition , 

management 

Break 

Nonacademic 

Son* of tM Abov» 



1 


♦ 


0 


4 


-O 


0 














2 


♦ \ 


0 


1 


0 


0 




4 










3 




0 


0 


2 


0 




♦ 










* 


♦ 


^0 


0 * 


0 


0 












1 


5 




0 • 


0 


0 




♦ 










6 




, 0 


0 


0 


0 




♦ t 










7 


♦ 


3 


0 


0 


0 



♦ 

Marginal Totals ♦ , 3 1 



0 
0 

0 
0 
0 
0 
*0 



0 
0 

0 
0 
0 



Marginal 
N^t a I • 






♦ 




0 


♦ 


0 




♦ 




0 


♦ 


1 




♦ 




l 




3 




♦ 




0 


♦ 


0 




♦ 




Q 


- ♦ 


1 








0 


♦ 


0 




♦ 




46 , 




49 










♦ 




4? 




54 




♦ 





Nominal Marginal 
agreement Agreement 



light's 
extension 
of KepPa 



1.000 
#66? 

0 #000 



i*39 



\ 



0t000 
1 #000 
#667 
OtOOO 
OtOOO 

%9St 



l.ooo 

^#654 
0#000 
f B8B 



Category Marginal AOreementl #*3B 
Cohents KePujl #SB4 



10.1 



ERIC 



i * 



/ 



Table 43 



phaia IV - IGE Evaluation Projact 
DetcrlPtlva StufJy - soring 1976 

M 

Reliability on central Content 
Obiarver 40 Export SO , 
School BOl Grade 2 

Heefc — o . Day J__. 

WDR5D* 



Category choteh Cat t 
bv Expart 60 # 



Category chotcn by pbterver 40 

1 2 3 v * 5 fi 1 



Marginal 
Totali" 



Marginal Totals ♦ 



49 



♦ 
♦ 



64 







♦ 
















♦ 




r 






wait 


1 


♦ 




0 


0 


0 


0 


0 • 




0 ♦ 




* 




transition 


8 


♦ 

♦ 

» 


0 


1 


0 * 


0 


0 


, o' 


0 


♦ 


1 ♦ 


1*000 


1*000 


1,000 


Hanegeroent 


3 


♦ 
♦ 


0 


0 


3 


0 


0 


0 


0 


♦ 
♦ 


3 ♦ 


1*000 


" ^»000 


1,000 






♦ 


















0 *t 




. o»ooo 




Break 


4 




0 


0 


.0 


0 


0 


0 


, 0 








Nonocademjc 


5 




0 


0 


1 0 


•1 


y 


0 


0 


. ♦ 
♦ 


1 ♦ 


0*000 


0*000 


p., 000 


Other Aeadamlc v 


6 


♦ 


0 ' 


0 


0 


0 


0 


0 


0 




0 > ♦ 








Nona o$ the Abova 


t 


♦ 


0 


0 . 


0 


0 


0 


m 0 


49 


♦ 


49 ♦ 


1*000 


1*000 


1.000 



Light's 

No« I na I Mar g I na | Ex ten* I on 
Agreement Agreement of* KaPPa 



category Marginal Agreement! *600 
Cohanta KaPpal #993 



102 



erJc 



V 



/ 



4 / 



Ca^gory choran 
by ExPart BO 



wait 

Tranti t Ion 
Manegamant 
^Braak 
htnacadamie 
Qthar Acadamlc ' 
NonV of tha Abova 

Marginal 



Table 44-* 

Phasa IV - XGE Evaluation Projact 
DatcrlPtlva study « spring 197& 



Ratability on General Contant 
Obtarver 10 Expart 60 
School sol orada 2 
Watk o ■ Day S 

^ DMP 



Cat i 
a - 



Catagory chotan by Obtarvar 10 
12 3*967 



Marginal 
Totals 



♦ 

1 ♦ 
, ♦ 

2 ♦ 
♦ 

3 ♦ 
" ♦ 

4 ♦ 

5 ♦ 
♦ 

7 ♦ 



0 
0 
0 
0 
0 
0 



0 
0 



o % ♦ 



3 

0 
9 
0 
0 
0 
73 




Totals ♦ 
♦ 

♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 



Nominal Marginal 
Agraamant Agraamant 



i667 



0*00.0 



,939 



,667 



UQht'l 
Extantlon 
o# KftPPt 



,658 



.,063 



.95* 



,3Jt 



Category Marginal Agraamant t *6S3 
_ Cohan»i Kappa! ,,347 ^ 



1 7 



9 

ERIC 



/ 



-103 



4> 



V 



Catagory cKoten 
by e*P«rt SO 



taal t 

Transition 
Panagamant 
iraak 

Nonaca*«mre 
Qthar AcadafMc 
Nor^a o# tha Abova 



Table 45 

Phaia IV,- IOE Evaluation Projact 
DaicrlPtlva Study - spring 1978 

Reliability on Ganaral Contant 
Obsarvar 20 Exptrt SO 
School 601 Orada 8 
Waak 0 Day 5 

_ ..„.._ OMP J . 



Cat* * 
* 



Catagory chotan by ObsarvaV 20 
1 3*867 



Marginal 
Total • 

i 

r ♦ 



. Noplnal Marginal 
Agra^mant Agraaman* 



■I 

2 ♦ 

3 *V 
♦ 

4 ♦ 
♦ 

8 ♦ 
♦ 

6 ♦ 
♦ 

7 ♦ 
♦ 



Marginal TotaT* + 



2 


0 


0 


0 


4 

0 


0 


1 


♦ 
♦ 


3 


0 


0 


0 


0 


p 


0 


0 




0 


0 


0 


0 


0 


0 


0 


2 


♦ 
♦ 


2 


0 


. 0 


0 






0 


0 


♦ 

♦ " 


o t 


0 


0 


0 


0 


0 


0 


0 


♦ 


0 

,°' 


0 


v 0 


0' 


0 


0 


0 


0 


♦ 


i 


0 


0 


0 


0 


0 


78 




73' 




.667 



0,000 



,966 



1,000 
0,000 

,673 



Extant lop 
of KaPPB 



,663 

0,000 



,666 



Cataoory Marelnal AorWaantl »6B6 
cohantt Kappas ,660 



4 I 



9 

ERIC 



104 



■ % 



Table 46 

Ph#se IV - IOE Evaluat Ion Project 
DescriPtJva Study - Spring 1978 

, Reliability on General Content 
' > Observer 30 Expert 90 

School eol orada 2 
Waak o Day S 

•DMP 



x 



Category chosen 
by Expert 90 



Malt 

Tram | tlon 
Management 
/Break 

9 

fconecademi e > 
Other Academic 
None o#^ the Above 

Marginal 



Cat. 



2 * 
♦ 

3 ♦ 
♦ 

4 ♦ 

5 +~ 
♦ 

6 ♦ 
♦ 

T ♦ 

♦■ 



Category cHoinn by Observer 30 



2 

0 
0 
1 
0 
0 
2 



0 
2 
0 
0 
0 
0 

1 



0 
0 
0 
0 
0 
0 



0 
0 

1 

o 

0 
0 



0 
0 
0 

0 

o 

0 
0 



0 
0 
0. 
0 
0 
0 
0 



* 0 
2 
0 
0 
0 
0 

73 



Marginal 
Totals 



♦ 
♦ 
♦ 
♦ 

♦ 

♦ 
♦ 
♦ 
♦ 
♦ 
♦ 
♦ 



h 

2 
0 
t) 
T6 



♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 

■ * 

♦ ♦ 

Totals ♦ 5 3 0 V 0 0 75 ♦ 8* 

♦ ♦ 

♦ ♦♦♦♦♦♦♦♦♦♦♦♦ ♦♦♦♦♦♦♦^♦♦♦♦♦♦♦♦♦4 ♦♦♦♦♦♦♦♦ ♦♦♦♦♦♦♦♦♦♦ 



Light's 

Nominal Marginal Extension 
Agreement Agreement t># KapPe 



1.000 

• 500 

• 500 



• 400 

• 750 



• 500 



it 61 



.907 



1.000 

«<»ai 

,494 
• 632 



Category Marginal Agreementl •»s9 
6ohen«e Kappa t ,ttl 



105 



ERIC 



Table, 47 ^ 

Phiie IV I6E Evaluation pro/ect 
Delcrl&JIVe Study - Soring 1978 

^Reliability on otntral C&ntent - 
ObierVar 10 Expert 90 
* School 801 Orade 2 

w.aeK _o D ay 5 

DMP 



egorvY chosen 
*bv Expert 90 

\ 

waVt 

Irani \ t Ion 
Management 
Breah 

Nonacademie 
Other Academic 
None o# the Above 



Cetegory ehoien by Observer, 10 



/ 



Cat, 
* 



Marginal 
Totaii 



Nominal 
Agreement 



Marginal 
Agreement 



Marginal 



\ 







► ♦♦♦♦<•♦♦♦♦ '.1 






1 




2 


0 


0 


0 


0 


0 


0 i 


> 2 .♦ 


1.000 


• 667 


z 




0 


2 


0 


0 


0 


0 


9 < 


► * ♦ 


• .600 


• 500 


























3 




0 




0 


0 


0 


0 


0 < 


► 0 •♦ 






























« 




1 


0 


0 


1 


0 


0 


0 i 


>i \ a ♦ 


• 800 , 


• 500 


5 




0 


0 


0 


0 


0 




0 i 


► 0 ♦ 








♦ 














1 








6 


♦ 


0 


0 


0 


o 


0 


0 


0 < 


► , 0 


'■mmrnmm » 




T 


♦ 


0 


0 


i 

0 j 


0 


0 


0 


76 < 




1.000 












l 


■ 


♦ 


3 


2 


0 ' 


1 


0 


0 


78 < 


► d* ' ♦ 

















Category Marginal Agreement I #860 
Cohtn«9 tuopet ,77 J 



UlOht'l 
extension 
of KaPPa 



1.000 
^ ,*B8 



1 •» 



\ 



100 



Ta 



48 



Phase IV - IGE Evaluation PKo/ect 
Descriptive S.tudy - spring 1978 

Re I l ab I I I ty on Pace 
Observer 10 'Expert 90 • 
School 801 Grade 2 
> ' Week 0 Day 4 
W0RS0 



\ 



Category* chosen 
by Expert 90 ' 



, Self 
other 
Nel ther 



category chosen** 
*by Observer 10 
Catt •• ^ Marginal 

« ■ 1-23 A opal's 

+ 4 + ++++++ ++++++++++ 



Nom | na 



Marg | na 



L I ght » s. 
Extens | on 



Agreement Agreement of Kappa 



30 



1 ♦ 

2 -f * . ' S 

3 ♦ 0 



0 
19 
0 



♦ 

0^s J 



0 
0 



30 

. 0. 



♦ 
♦ 



1 .000 
.792 



.857 
• 792 



1,000 
# 679 



Marginal Total* * 35 19 

♦ 

* ♦+'+*++♦♦ * 



♦ 
♦ 



54 



♦ 



/ 



category Marginal Ao/oewentl >824 



r 



coh«n*s Kappal 



809 



-J 



107 



108 



Table 49 



Phase IV -, IQE, Evaluat Ion Project 
Descriptive Study - SPrln© 1978 

Re' I lab I 1 1 ty on pace 
Observer 20 Expert 90. 
School B01 Grade 2 
■ Week 0 Day 4 
* WDRSO 



Category chosen 
by Expert 90 



cat. 



Category chosen 
by Observer 20 

1 2 3 

♦♦♦♦♦♦♦♦♦♦♦♦+♦♦♦♦♦+ 



Marol na I 
Totals 







♦ 
















Self 


1 




27. 


2 


' 1 < 


► 30 ♦ 


• 900 


• 938 


,7^5 


Other 


2 


♦ 


5 


17 


2 i 


► 24 ♦ 


.708 


.792 , 


,550 


♦ 

Net ther 


3 


♦ 


0 


0 


0 i 


► 0 ♦ 




0,000 


\ 


* 














Marginal 


Total $ 


♦ 


32 


19 


• 

3 < 


► 54 ♦ 









Norn | na I 
Agreement 



Marginal 
Agreement 



Ciahtts 
Extens ion 
of Kappa 



Category Marginal, Agreementl ,576 
Cohen's Kappal ,640 



Table ,50 

Phase IV - IOE Evaluet Ion Project 
Descriptive Study - spring 1978 

j 

Re I | eb I I I ty on pace 
Observer #30 Expert 8p ' 
School 801 Grade 2 
■ WeeK o Day 4 
WORSD 



Category chosen 
by Expert 80 



Cat, 



Cateaory chosen 
Observer 30 



r 



1 



Marginal 
Tota I s 



NQm | na I 
Agreement, 



Marg | na I 
Agreement 



Light«s 
Extens I on 
of Kappa 





♦ 




H 




♦ 








Se 1 f 


1 ♦ 


33 


0 , 


1 




34 


♦ v .97l ' 


1*000 




■ ♦ 








♦ 






Other 


? ♦ 
♦ 


0 


15 


, o. 


♦ 
+ 


15 


♦ 1.000 


• 833 


Neither, 


3 ♦ 


1 


3 


7 


♦ 

♦ * 


11 


♦ .636 


• 727 




♦♦♦♦♦♦♦♦ 
















♦ 








♦ 




• • 




Marginal Tota 

r 


s ♦ 
s* 


34 

1 


18. . 


8 


♦ - 


60 




i.i 



• 938 
i»000 

• 580 



category Marginal Agreement! «854 

i 

Cohen » • Kappas ,856 



ill 



112 



Table 51 

Phase IV • JOE Evaluation Project 
Descriptive study - spring 1978 

Pel labial I ty on Pace 

Observer 40 Expert 80 

School 801 Grade 2 

week 0 Day 4' 
wd&SD 



:< 



O 



Category chosen 
by Expert .80 



113 



Self 
Other 
Ne | ther 



Category chosen 
by Observer 40 



Cat t 
H 



1 



3 



-Maro I na I 
Total s v 



1 + 33 0 1 ♦ 34 

♦ ♦ 
*2 ♦ 0 15 0 ♦ 15 . 

♦ ♦ 
3 '♦ 1 2 8 ♦ 11 

♦♦♦♦ ♦ ♦♦♦♦ ♦♦♦>♦♦♦♦ ♦♦♦♦♦♦♦♦♦+♦ 

♦ ♦ 
Marginal Totals ♦ 34 17 9 ♦ 60 

♦ - ' ♦ 



Nom| na I 
Agreement 



.971 
ItOOO 
• 727 



Marginal ' 
Agreement 



1.000 

• 882 

• 818 



Light»s 
Extens | on 
of Kappa 



,932 
1,000 
,679 



f 



114 



category Marginal Agreement! «900 
Cohan's Keppal •885 



9 

ERIC 



Category cooler* 
by Expert 80 



Table 5*2 

Phase IV - *!0E Evaluation Project 
Oeicr^gtlve Study - SPrlnd 1976 

Re I lab 1 1| ty on Pace r- 
Observer \0 Expert BO 
School 801 Grade 2 
w^ek 0 Day 4 

OMP 



Cateqory chosen ^ 
by Observer. 40 V 

Cat, Marginal Nominal 

* 1 • 8 3 .Totals', Agreement 

♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 



.794 

• 871 

• 89S 







♦ 








♦ 




♦ 


Self 


1. 


♦ . 


87 




• 2 




34 


\ 














♦ 






other 


_2 


♦ 


4 


27 


0 


♦ 


31 








♦ 












■ ♦ \ 


.Nel ther 


3 


♦ 


1 


1 


17 


4 


19 


♦ 

i 






♦ 








♦ 




♦ 




i 


♦ 








♦ 






Marginal 


Totals 


♦ 


32 


33 


19 




84 





Marginal 
Agreement 



.941 
• 939 
1.000 



U/£ht»s 
Extens ion 
o* Kappa 



,667 
,787 
,864 



115 



category Marginal Agreements ,960 
Cohen«s Kappa) ,762 



t * 



116 



Category chosen 
by Expert 80 



Table 53 

Phase IV • IGE Evaluation Project 
Descriptive Sturdy - spring- 1978 

Pe l lab.l I I ty\on Pace 
Observer 20 . Expert '80 
School '801 Grade 2 
Week 0 M>ey 4 
A 'DMP 



Cat, 



117 



Cateaory chosen 
by Observer 20 

\ 2 3 







♦ 
















Self 


1 


♦ 


30 


•4 

3 


1 ♦ 


34 


♦ 


• 882 


• 89S 


* 














♦ 






other 


2 


♦ 


... T 


24 


0 ♦ 


31 


♦ - 


J774 


.871 






♦ 










♦ 






Nel ther 


3 


♦ 


1 


0 


18 ♦ 


19 


♦ 


• 947 


1«000 


< 




♦ 










♦ 




* 






+•4 +++ + + + + + + + + ♦♦♦♦♦♦♦ 


♦ ♦♦♦♦♦♦♦<♦ 






Marginal 




♦ 










♦ 






Tota 1 s ' 


♦ 


38 


27 


19 ♦ 


84 












♦ 










♦ 







Marginal , Nomfnal 
Totals Agreement 

♦♦♦♦♦♦♦♦♦ 



r 



Marg I na I 
Agreement 



o 



L|g*tts 
Extehs I on 
of Kappa 



• 785 

• 667. 

• 932 

, lis 



r 



( 



+ ++++ ++++++++ ♦ ♦++ 



category Marginal Agreement^ «922 
Coh'en» s Kappa I .779 



o 

ERIC 



J. 

Table 54 



Phase IV • IOE Evaluation Project 
Descriptive stud* - spring 1978 

Reliability on pace-'* j 
• Observer 30 Export 90 

School B0\ Grade x j T$ 

Week 0 Day 4 
DMP 



Category chosen 
by Expert 90 



ser* 



y 

Other^ 
Neither 



Cat. 



Category chosen 
by-' Observer ' 30 



1 2 3 



Marginal Nominal Marginal 
Totals ' Agreement N Agreement 





♦ 








♦ 




♦ 


4 


1 


♦ 


46 


I. 


1 


♦ 


48 




• 958 




■*> 












♦ 




2 




S 


14 


3 




22 


♦ 


♦ 636 












♦ 








3 


• ♦ 


\ 5 


0. 


3 




8 


♦ 


• 375 












♦ 












++ + + + + ♦ ♦ ♦♦♦♦♦♦ ♦♦♦♦♦♦♦♦♦ 














♦ 




♦ 




s 




5* 


IS 


7 


♦ 


78 


♦ 






♦ 

















• 857 

• 682 

• 875 



Llghtts 
Extension 
of Kappa 



• 652 

• S5tf 

• 313 



/ • 



9 

ERIC 



category Marginal Agreement! #805 
Cohen»s Kappa I 9 611 



6 

U) 



119 



/ 



Table 55 



Phase iv - 1.0 E Evaluation Project 
Descriptive study - spring 1978 

Re I | ab I I I ty on Pace 
Observer 40 Expert 90 v 
SLchool 801 Grade' X 
4 Wee* 0 Day 4 . 
DMP ' 



o 



121 



Category chosen 
by Expert 9.0 



Cat, 



Category chosen 
by Observer 40 



1 



Marginal 
Tota I s 

















♦ 




Sel f 

t 


1 


♦ 
♦ 




• 4 


0 


♦ 

♦ f 


, 48 ♦ 


« .917 


Other 


2 


♦ 


4 


18 


0 


♦ 


22 ♦ 


• 818 




♦ 












Nel ther 


3 


• 


1 


0 


7 ' 


♦ 


8 h ♦ 


» .875 














♦ 








♦ 






Marginal 


Total s 


♦ 


**. 


22 


7 


♦ 
♦ 


78 ♦ 
♦ 


A- 



Nominal Marginal 
Agreement Agreement 

.980 
1.000 
,875 



Category Marginal Agreement! »952 
Cohen »s Kappa I .780 



Llghtis 
Extens ion 
of Kappa 



• 77* 
.747 

• 863 



i 2> *v 



9 

ERIC 



Table 5& 



Phase XV • IGE Evaluation Project 
Descriptive Study - spring 1978 

Re I I ab I 1 1 ty on Pace . 
Observer 10 .Expert 9o 
School 90} -- : <»rad* 5 
Week 0 Day 5 
DMP 



Category chosen 
by Expert 90 



Cat, 



Category chosen 

b y~t)b rerver 10 



1 2 3 



Marginal Nominal 
Total s * Agreement 



Marginal 
Agreement 



Llght»s^ 
Extens Ion 
of Kappa 







♦ 












L ♦ 








Self 


1 




25 


0 


2 


♦ 


27 


♦ 


• 926 


• 926 


,556 






♦ 




















Other 


2 




0 


0 


0 


♦ 


0 


♦ 












♦ 








♦ 




+ 








Ne| ther 


3 




0 


0 


3 


+ 


3 




1.000 


• 600 


1,000 






♦ 
























444444444444444++**+*+++4*+4+ 












♦ 








♦ 




* 








Marginal 


Total $ 




25 


0 


5 




30 


♦ 












♦ 








♦ 




♦ 









4 44 4,444444444.44. 44- ♦♦♦♦♦♦♦♦♦♦♦♦ 



category Marginal Agreement I ,763 
Cohen's Kappat ,714 



123 



124 



Table 57 



o 



Phase XV - IGE Evaluation Project 
Descriptive Study - spring 1978 

Rel labl I I ty on Pace; 
Observer 20 Expert 90 
School* 801 Grade 5 
Wejsk 0 Day 5 
DMP 



-Category chosen 
by Expert 90* 



Cat, 



Category chosen 
by Observer 20 



MarQlnal Nominal ' Marginal 
Totals Agreement Agreement; 



Llghtts 
Extens I on 
of Kappa 



Self 
Other 
Net ther 



1 
2 
3 



Marginal Totals 



\ 



27 
0 
1 

28 



0 

0 
0 



0 
0 
2 



♦ 
♦ 

♦ 
♦ 

♦ 



27 
0 

■ 3 
30 



♦ 
♦ 
♦ 
♦ 

♦ 

♦ 
♦ 



uooo 



,667 



964 



667 



1,000 



643 



category Marginal Agreementl ,8lS 
Cohan's Kappas ,783 



126 



Table 58 

Phase XV - llOE. f valuat Ion Project 
DescriptlveVStudy - Spring 1978 

Re I lab I I I ty on pace 
Observer 30 Expert Bo 
School 601 Grade 5 
Week 0 Day 5 
OMR 



Category chosen 
by Expert 80 



Cat. 
* 



Category chosen 
by Observer 30 



Mero Inal 
Total s 



1 2 3 



Nominal 
Agreement 







♦ 
















Sell 


1 


♦ 
♦ 


11 


0 


0 


♦ 
♦ 


11 1 


> 1,000 


1*000 


Other 


2 


♦ 
♦ 


0 


12 


0 


♦ 
♦ 


12 i 


► 1.000 


1*000 


Nt| ther 


3 


♦ 


0 


0 


7 


♦ 


7 i 


> 1*000 


1*000 



♦♦♦♦+♦+♦♦♦+♦++♦♦♦♦♦+♦++♦++++ 
♦ ♦ 
Marginal Totals ♦ 11 12 7 ♦ 30 

++++«++*+++++++++++ ♦♦♦♦♦♦♦♦♦ 




^ght»s 
Marginal Extension 
Agreement of Kappa 



1,000 
1,000 
1,000 



category Marginal Agreement! 1,000 
Cohen»* Kappat 1,000 



9 

ERIC 



127 



128 



o 



V 



Table 59 

Phase tv • IGE Eva* l uat ) on Projact 
Oascrlptlva study - spring 1978 

\ 

Rel labl I I ty on paca 
Observer 40 Expert 80 
' School 601 Grade S 
week 0 -Day, 5 
OMP 



Category chosen 
by Expert 80 y 



Cateoory chosen 

by .Observer 40 f 

Cat* Marginal 

* I ' 2 3 Totals 



Nominal 'Marginal 
Agreement Agreement 



Llght»s 
Extens I on 
of Kappa 







♦ 




















Sell 


1 


♦ 


11 


0 


0 


,♦ 


11 


♦ 


1*000 


1*000 


1*000 






♦ 








♦ 




♦ 








Other 


2 


♦ 


0 


12 


, 0 




12 


♦ 


1*000 


• 923 


1*000 






♦ 












♦ 








Net ther 


3 




0 


1 


6 


. ♦ 


7 


♦ 


• 857 


• 857 


• 821 










* 




♦ 
















♦♦♦♦♦♦♦♦♦ ♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 
























♦ 








Marginal 


Total s 




1; 


13 


6 


♦ 


30 


♦ 




















♦ 




♦ 









category Marginal Agreement) «927 
Cohen's Kappa) ,948 



1 3 3 



130 



> . Table 60 

Phase lv - IGE Evaluation Project/ 
Oescriptlve Study - sprlno 1978 

Re I | ab I 1 1 ty on pace 
Observer 10 Expert 9o 
School 801 Crade 2 
Week 0 Day 5 
WDRSD 



Cateoory chosen 



Category chosen 
by Expert 90 * 


Cat 
u 


• 


by 

1 


Observer 20 
2 3 


Marginal 
Total s 


Nominal 
Agreement 


Marginal 
Agreement 


Llghtts 
Extent loi 
of Kappa 








►+'♦♦+++++♦ 






* 


Self 


1 




0 


S 1 H 


► 6 ♦ 


0,000 


0.000 


0,000 


Other 


2 


♦ 


0 


46 1 i 


► 47 ♦ 


• 979 


• 922 


,617 


Ne| ther 

v. 


3 


♦ 


0 


0 1 < 


" 1 ♦ 


1*000 


• 333 


1, 000 






.A. 










Marginal 


Total • 




0 


51 3 < 


• 54 ♦ 









category, Marginal Agreement I ,4is 
Cohen's Kappat ,267 



131 



132 



Table 61 



Category chosen 
by Expert 90 



Phate iv - ige Evaluation Project 
Descriptive Study • Spring 1970 

• * 

Pel lab I 1 1 ty on Pace 
Observer 20 Expert 90 
School 801 Grade 2 
weak o Day 5 
WDRSO 



Category chosen 
by Observer 20 



Cat, 
a 



Marginal 
Totals 



1 2 3 







♦ 










J? ♦ 


Self 


1 




0 


5 


1 


♦ 


6 * 
















♦ - 


Other 


2 


♦ - 


0 




* 2 


♦ 
















♦ 


♦ 


Neither 


3 


♦ 


0 


1 


0 


♦ 


1 ♦ 
















♦ 






♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦ ++4 






♦ 










♦ 


Marginal 


Total s 


♦ 


0 


si 


3 


♦ 


54 ♦ 










• 






♦ 




Nomi 
Agree 



O.ooo \ 



Marginal 
Agreement 



0.000 
• 922 
.333 



♦♦♦♦♦♦♦♦♦♦♦ ♦♦♦♦♦♦♦ ♦♦♦♦♦♦♦♦ 



Category Marginal Agreement) .418 
Cohen *s Kappal .058 ' 



Table 62 

Phase XV - IOE Evaluation Proj\set 
Descr |pt I ve Study - spp Ing 1978 

Re I | ab I I I ty on Pace 
Observer 30" Expert 8o 
School 801 Grade 2 
week 0 Day 5 
WDRSD > 



v ; 



Category chosen 
by Expert 80 



Cat. 
u 



Category chosen 
by Observer 31 



Marg I na I 
Tota.ls 



♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 

♦ ♦ ♦ 
Self 1 ♦ .41 4 a ♦ , ♦ 

♦ ♦ ♦ 
Other Z ♦ 0 0 2 ♦ 2 ♦ 

Neither r 3 ♦ .1 1 3 5 ♦ 

* ♦ ♦ " ♦ 

♦ > . ♦ ♦ 
Marginal' Totals + 42 S 7 ♦ 54* ♦ 

♦ ♦ ♦ 

4 + + + + .i. + + + + + + + + + + 4- + + + + + + + + + + + + 



Nominal 
Agreement 



• 872 
0.000 

• 600 



Marginal 
Agreement 



• 89* 

• 400 

• 714 



Llghtts 
Ex tens | on 
of Kappa 



• 426 
• •102 

• 540 



/ 



category Marginal Agreement! «669 
» Cohan's Kappat ,398 



135 



r 



/ 



136 



Table 63 

Phase IV - IOE Evaluation Project 
Descriptive Study - SPrlnQ 1978 

Re 1 1 ab 1 1 1 1 y on Pace 
Observer 40 Expert Bo 
School 801 Grade 2 
Week o Day 5 
WDRSD 



Category chosen 
by Expert 80 



Cat, 
u 



Ceteaory chosen 
by Observer 40 



Mara I na I 
Tota I s 



Nominal 
Agreement 



Marg | na I 
Agreement 



Light »s 
Extens I on 
of Kappa 







♦ 








♦ 




Self 


1 


* 


47 


0 


0 


♦ 


47 


















Other 


2 




0 


2 


0 




2 






♦ 












Ne| ther 


3 


♦ 


.0 


0 


5 




5 






♦ 








♦ 
















♦ 




Marginal 


Total s 




47 


2 


5 




54 






♦ 








♦ 





♦ 
♦ 

♦ 

♦ 



1*000 
1*000 
1*000 



1.000 1,000 
1*000 1,000 
1*000 1*000 



category Marginal Agreement! 1*000 
Cohen's Kappa j 1*000 

13 d 



137 



9 

ERIC 



( 



\ 



Category chosen 



Table 64 

l 

Phase XV • I6E Evaiuat lon^project 
Descriptive Study • spring 1978 

Re I lab) 1 1 ty on Pace 
Observer 10 Expert 80 
School 801 Grade 2 
Week 0 Day S 
OMP 



Category chosen 
by Observer 10 



Cat 



Marginal 



Expert 80 


n 




1 


2 


3 




Totals 




» 


44444444444444444444444444444 






4 








4 




self 


l 




31 


1 


0 


4 


32 4 






♦ 








4 




Other 


2 


4 


0 


36 


5 


4 


41 4 






♦ * 








4 




Net ther 


3 


4 


0 


2 


3 


4 


5 4 






4 








4 




* 




4444 44444 44444444 444 444 44 44 44 






4 , 








4 




Marginal 


Total s 


4 


31 


39 


8 


4 


78 4 






4 








4 





4444444444444444444 44-4444 44 4 



Nominal Marginal 
Agreement Agreement 



• 969 
.878 

• 600 



.969 

• 951 

• 625 



Llghttt 
Extens ion 
of Kappa 



.948 
.756 
• 554 



S. 



/ 



category Marginal Agreements .848 

* i 

Cohentt Kappa) .819 



139 



110 



H 



/ 



. Table 65 

Phase !v - IOE. Evaiuat Ion Project 
Descriptive study»- spring 1978 

Re I lab 1 1 I ty on Pace** 
Observer 2o Expert 80 
•° School 801 Grade ? 
s Week o Day 5 

OMp . 



Category chosen 
by Expert 80 



Self 

Other 

Neither 



Cat* 
n 



Marginal Totajs ♦ 

♦ 



Cateqory chosen 
by Observer 20 



1 



32 



42 



Marginal 
Total s 

>♦♦♦«>♦ 



♦ 
♦ 



78 



Norn | na | 
Agreement 



Light* j 
Marginal Extension 
Agreement /rf Kappa 













♦ . 










1 


♦ 
♦ 


32 


0 


0 


♦ 


32 k 


► 1*000 


1*000 


1,000 


2 


♦ 


0 


39 


2 






*931 


• 976 


• 89* 




♦ 
















3 




0 


3 


2 




5 4 


• 400 


• 800 


• 368 




♦ 








♦ 











Category Marginal Agreement! »92s 
Cohen's Kappat ,862 



ill 



J42 



ERIC 



Table 66 

Phase IV - IGE Evaluation Projact 
Descriptive Study -spring 1978 

Re I I ab I I I ty on Pace 
Observer 30 Expert 90 
School 601 Grade 2 
week 0 Day 5 
OMP 



Category chosen 
by Expert 90 



Cat. 



Category chosen 
by Observer 30 



1 



Marginal Nominal Marginal 
Totals" Agreement Agreement 



Llght*s 
Extent ion 
of Kappa 



self 
Other 
Nel ther 













♦ 






• 98* 


1 


♦ 


56 


6 


1 


♦ 


65 4 


• 862 












♦ 








2 


♦ 


8 


1 


2 




11 * 


• 091 


• .818 




♦ 








♦ 








3 


♦ 


0 


0 


6 


♦ 


8 4 


1«000 


.727 





















64 



Marginal Totals *■ 



-A 



84 



.418 
• .018 
1.000 



category Marginal Agreementl .843 
Cohen's Kappal .411 



143 



144 



.1 



s * 



\ 



\ 



Table 67 

Phase IV - XOE Evaluation Project 
Descriptive study • spring 1978 

Re I | *b | | I ty on Pace ' ■ 
Observer *f0 Expert 9o 
Schoo.l 801 Grade 2 
Week 0 Day 5 
OMP 



Catenary c+\oten 
by Expert 90 



Cat* 



Category chosen 
by Observer 10 



Marginal 
Total s 



1 2 3 







♦ 








♦ 










Self 


1 


♦ 
♦ 


63 


S 


0 


♦ 


65 


• 969 


• 915 


• 801 


Other 


a 


♦ 

♦ . 


8 


2 


1 


♦ 


11 ♦ 


• 182 


• 364 

i 


• 141 


Neither 


3 


♦ 


0 


0 


8 


♦ 
♦ 


8 ♦ 


1*000 


• 889 


uooo 






+ ++ + + ++ ++++++++++ ++++++++ 




















♦ 










Marginal 


Total s 


♦ 


71 


4 


9 


♦ 


84 ♦ 









♦ ♦ 

+++++++++++++++++++++ ♦ ♦♦♦♦♦<► 



Nominal 
Agreement 



Marginal 
Agreement 



Light* s 
Extens |on 
of Kappa 



category Marginal Agreementl «723 
Cohen's Kappat «603 



14 j 



i * * 



ERIC 



Table 68 

Phase XV • IOE Evaluation Projtct 
Descriptive study • Spring 1978 

Reliability on Grouping 
Observer 10 Expert 90 
School 601 Grade 2 
week o Day 4 
WDRSO 



Category chosen 
by Expert 90 



Category chosen by Observer 10 



Cat, 
* 



hargine I 
Totals 



tal 

Ag/eement 



Llght»s 
Marginal Extension 
Agreement of Kappa 



Marginal Totals 



Individual 




1 ♦ 


20 


0 


0 


0 


0 < 


► 20 4 


► l.OoT"" 


• 645 


1,000 


Pair 




2 ♦ 


2 


2 


1 


0 


0 i 


► S 4 


► .400 


• 400 


,377 


Smaj | Group 


(3-fl) 


3 ♦ 


5 


0 


7 


1 


0 4 


► 13 < 


• 538 


• 615 


,458 


Large Group 


t>B) 


4 ♦ 


4 


0 


0 


12 


0 4 


► 16 4 


• 750 


• 813 


,671 


None of the 


above 


5 ♦ 


0 


0 


0 


0 


0 < 


* 0 « 









31 



8 



13 



++++++++ 



♦♦♦♦♦♦♦♦ 
54 



Category Marginal Agreement! ,618 
Cohents Kappai ,644 



147 



148 



Table 69 

Phase Iv - 1GE Evaluation Project 
OescrlPtlve Study - Spring 1978 

u 

Re I |ab I I Ity on Grouping 
Observer 20 Expert 90 
• , School 801 Grade, 2 
Week 0 bay 4 
WDRSD 



Category chosen 
by Expert 90 



Category chosen by observer 20 
Cat# Marginal 
'# l 2 3 4 S Totals 



♦♦♦♦♦♦♦♦ 



Marginal Totals 



Individual 




1 ♦ 


19 


0 


- 0 


0 


1 < 


f 20 ♦ 


#950 


• 741 


,900 


Pair 


• 


2 ♦ 




1 


0 


0 


6 < 


► 5 ♦ 


• 200 


• 400 


,169 


Small GrouP 


<3-8> 


0 ▲ 

3 ♦ 


1 


1 


10 t 


0 


1 i 


► 13- ♦ 

i 


• 769 


•Ye 9 


.717 


lerge Group 


OB) 


4 ♦ 


3 


0 


0 


12 


I .* 




• 750 


• 75o\ 


,679 


None of the 


above 


5 ♦ 


0 


0 


0 


0 


0 « 


► o ♦ 




" 0,000 





27 



'10 12 



, Nominal 
Agreement 



Marginal; 
Agreement 



♦♦♦♦♦♦♦♦ 
54 

♦♦♦♦♦♦♦♦ 



Light's 
Extens Ion 
of. Kappa 



14 'J 



cetegcry Marginal Agreement i , l -32 
Cohents Keppat ,683 



150 



Table 70 



Phase IV • IGE Evaluation Project 
Descriptive study - spring 1978 

Re I |abl I I ty on Grouping 
Observer 30 Expert 80 
, School 801 Grade 2 
week 0 Day 4 
WORSO 



Category chosen by observer 30 



Category chosen 
by Expert 80 



Cat, 
n 



l 



Marginal Totals ♦ 

♦ 



29 



18 



8 



♦♦♦♦♦♦♦♦♦ 

60 ♦ 
♦ 

♦♦♦♦♦♦♦♦♦ 



Xnd 1 v | dua 1 




1 




28 


2 


0 


0 


1 < 


" 31 


♦ 


• 903 


,935 


,813 


Pa 1 r 




2 


♦ 


0 


3 


0 


0 


0 4 


► 3 




1*000 


• 600 


1,000 


Sma I | Group 


(3*8) 


3 


♦ 


0 


0 


0 


0 


0 4 


► 0 


♦ 








Large Group 


(>8) 


4 


♦ 


0 


0 


0 


is 


0 4 


► 15 




1*000 


• 633 


1,000 


None of the 


above 


5 


♦ 


1 


0 


0 


3 


7 4 


► 11 


♦ 
♦ 


♦ 636 


.727 


.580 



Marginal Nominal 
Totals Agreement 

♦♦♦♦♦♦♦♦♦ 



Marginal 
Agreement 



Light's 
Extens I or 
of KaPPa 



Category Marginal Agreementi .774 
Cohen's Kappa! ,820 



ID* J52 



Table 71 

Rhase IV • IGE Evaluation project 
Descriptive Study * Spring 1978 

Reliability on Grouping 
Observer 40 Expert 80 
School 801 Grade 2 * 
week o Day 4 
WDRSO 



o 



Category chosen 
by Expert 80 



Category chosen by Observer 40 



Cat • 



Margjna I 
Totals 



Nominal 
Agreement 



Marginal 
Agreement 



Llght»s 
Extent I on 
of Kappa 



Individual 




1 


♦ 
♦ 


27 


3 


0 


0 


1 < 


► ,31 < 


► #871 


,903 


Pair 




2 


♦ 


0 


3 


0 


0 


0 < 


► 3 i 


> 1*000 


• 500 


Sma I | Group 


(3-8) ; 


3 


♦ 


0 


0 


0 


0 


0 i 


► 0 < 




'«««S)«I 

mm 


Large orouP 


C>8) 


4 


♦ 
♦ 


0 


0 


0 


15 


0 i 


► 15 < 


1,000 


• 882 


fcone of the 


above 


5 


♦ 
♦ 


1 


0 


0 


2 


6 < 


► u < 


• 727 


• 818 



Marginal Total* ♦ 



28 



% 0 



17 



♦♦♦♦♦♦♦♦ 



60 

♦♦♦♦♦♦♦♦ 



i 



,758 



1,000 

1.000 
,679 



Category Marginal Agreement i ,776 
Cohen* s Kappat »822 



154 



Table 72 

Phaie Iv - I6E Evaluation Projtct 
Oetcrlotlve Study - Spring 1978 

Reliability on Grouping " 
Observer \0 Expert 80 
School 801 Grade 2 
Weak 0 Day 4 

OMP 



Category chosen 
by Expert 80 



Individual 
Pair 

Smal | orouP (3-8) 
Large Group <>8) 
None of the above 



Category chosen by Obierver 40 
Cat. Marginal 
# I 2 3 4 5 Totali 



1 ♦ 20 
♦ 

2 ♦ 2 
♦ 

3 ♦ 2 
♦ 

4 ♦■ 0 
♦ 

5 ♦ 1 

Marginal Total* ♦ 25 

♦ - 



0 
6 

0 
0 
0 



11 



23 



19 



♦ 



84 



Nominal 
Agreement 



Marginal 
Agreement 



Llght»i 
Extens Ion 
Of KapPa 



2 


2 


2 


♦ 


26 « 


,769 


,962 


,671 


0 


1 


0 




9 « 


,667 


,667 


,641 


9 


0 


0 


♦ 
♦ 


11 < 


,818 


1,000 


.791 


0 


19 


0 


♦ 
♦ 


19 < 


1,000 


,826 


1,000 


0 


1 


17 


♦ 
♦ 


19 < 


► ,895 


1,000 


,864 



category Marginal Agreementi ,R91 

Cohen** Kappal ,799 & 

156 

" 1 55 



Table 73 

Phase XV - XGE Evaluation Project 
Descriptive Study - Spring 1978 

Re I I ab I l | ty on Grouping 
Observer 20 Expert 80 
School 801 Grade 2 
week 0 Day 4 
DMP 



Category chosen 
by Expert 80 



Individual 
Pair 

Smal I Group (3-8) 
Large Group <>8) 
None of the above 



i 



Category chosen by Observer 20 
Cat* . Marginal Nominal 

u 12 3^5 Totals Agreement 



Marginal 



Light's 
Extens I on 



Agreement of Kappa 



1 ♦ 
♦ 

2 ♦ 
♦ 

3 ♦ 
♦ 

4 t 

5 ♦ 
♦ 



Marginal Totals ♦ 

♦ 



20 
1 
2 
0 
1 

24 



0 
7 
0 
0 



4 
1 
9 
0 
0 



1 
0 
0 
19 
0 



14 20 



1 
0 
0 
0 
18 

19 



♦♦♦♦♦♦♦♦♦ 
♦ 

26 ♦ 
♦ 

9 ♦ 
♦ 

11 ♦ 

19 ♦ 
♦ 

19 ♦ 

+ + + + + + + + 4. 

♦ 

8* ♦ 



,769 
#778 

• 818 
1*000 

• 947 



• 923 

• 778 

• 78C 

• 950 
1*000 



,677 
,758 
,782 
1,000 
,932 



r- "1 



Category Marginal Agreement) ,887 
Cohen's Kappa) ,831 



158 



Table 74 

Phase IV • XGE Evaluation Project 
Descriptive Study - Spring 1978 

Re i | ab I I I ty 
Observer 30 
School 301 
Week o 



on Grouping 
Expert 90 
Grade S 
Day 4 



DMP 



Category chosen 
by Expert 90 



Cat* 



Category chosen by Observer 30 



Marg | na I 
Total s 



Norn I ne I 
Agreement 



Marginal 
Agreement 



Light •$ 
Extens I on 
of Kappa 



Marginal Totals 



Individual 




1 ♦ 


36 


1 


. 0 


0 


I * 


► 38 




.947 « 


,745 < 


,846 


Pair 




2 ♦ 


3 


S 


0 


0 


0 4 


f 8 


♦ 
♦ 


,625 


• 750 


,594 


Smai j Group- 


(3-8) 


3 ♦ 


7 


0 


6 


' 1 


t 4 
0 4 


► 14 


♦ 

■ ♦ 


• 429 


• 429 


,381 


Large Group 


(>8> 


4 ♦ 


0 


0 


0 


7 


3 4 


► 10 


■ ♦ 
♦ 


.700 


• 800 


,666 


None of the 


above 


s ♦ 


5 


0 


0 


0 


3 4 


* 8 




• 375 


• 875 


,313 



51 



a 



♦♦♦♦♦♦♦♦♦ 
♦ 



+++++++++ 
♦ 

7.8 ♦ 
♦ 



1.59 



Cetegory Marginal Agreement! ,720 
Cohems Keppat ,578 



M 



9 

ERIC 



160 



Table 75 



Phase Xv • IGE evaluation Project 
Descriptive Study «• Spring 1978 



4> 



R«l labl I I ty 
Observer 40 
School eoi 
Wee* 0 



on Group I no 
Expert 90 
Grade 5 
Day 4 
DMP 



Category chosen by Obterver 4.0 



Category chosen 
by Expert 90 



Cat, 
a 



♦s 



Individual 


1 


4 


35 


2 


0 


I 


0 


Pair 


2 




8 


6 


0 


0 


0 


Sma| | Group (3-8) 


3 


♦ 


6 


0 


6 


0 


0 






♦ 












Large orouP 1>8) 


4 


♦ 


0 


0 


0 


10 


0 


None of the above 


5 


4 


1 


0 


0 


0 


7 






♦♦♦♦ 














♦ 








• 




Maro! rva! 


Total « 


♦ 


44 


8 


8 


11 


7 



Marginal 
Totals 



♦ 

♦ 
♦ 
♦ 

♦ 

♦ 



38 
8 
14 
10 
8 

78 



Nominal 
Agreement 



Marginal , 
Agreement 



Llght»s 
Extens I on 
ot Kappa 



• 921 


• 864 


,819 


• 750 


1.000 


,T21 


• 571 


• 571 


^\S22 


1*000 


,909 


1,000 


• 875 


,875 


,863 



category Marginal Agreement! ,844 

ri Cohen»s Kappal ,770 1 u ^ 

Jul i 



ERIC 



Table 76 

Phase' Iv • IGE Evaluation project 
Descriptive Study - Spring 1978 

Reliability on Grouping 
Observer 10 Expert 90 
School BO 1 Grade 5 
week o Day 5 
DMP 



Category chosen 
by Expert 90 



Category eho*en by Observer 10 



Cat, 
# 



5 



Marg | na | 
Total s 



Norn I na I 
Agreement 



Marg | na | 
Agreement 



Light's 
Extent I on 
of 



Individual 


1 




22 


0 


0 


0 


2 


Pair 


2 


<¥ 
♦ 


0 


1 


0 


' 0 


0 


Small Group <3~fi) 


3 


♦ 


1 


0 


0 


0 


0 


Large GrouP^OO) 




♦ 
♦ 


1 


0 


0/ 


0 


0 






♦ 






J 






None of the above 


5 


♦ 


0 


0 


0 


0 


3 



♦ 
♦ 



24 



Marginal Totals ♦ 



♦ 
♦ 

1 ♦ 
♦ 

I ♦ 
i ♦ 

.3 ♦ 
♦ 

♦♦♦♦♦♦♦♦♦ 
♦ 

30 ♦ 
♦ 



1.000 
0*000 
0,000 
1,000 



1,000 
1,000 
0,000 
0,000 
,600 



,583 
1,000 
0,000 
0,000 
1,000 



9 



cateoory Marginal Agreement! ,520 
Cohen 1 1 Kappa t ,610 



163 



184 



Table 77 



Phase IV - IGE Evaluation Project 
DatcrlPtlve study - Spring 1976 



Reliability on Grouping 
Observer 20 Expert 90 
School 601 Grade 5 
Weak 0 Day 5 
OMp 



Category chosen 
by Expert 90 



Category chosen by Observer 20 



Cat. 

M 



Marg | na I 
Tota I s 



Nominal 
Agreement 



"Harg|na| 
Agreement 



Light »s 
Extens Ion 
o* Kappa 



Individual 


1 


+ 


24 


0 


0 


0 


0 


♦ 24 ♦ 


1 #000 


,857 


1,000 


Pair 


2 


♦ 
♦ 


1 


. 0 


0 


0 


0 


♦ 1 ♦ 


0,000 


0,000 


0.000 


Sma | | croup (3*8) 


3 


♦ 


1 


0 


0 


0 


0 


♦ 1 * 


o#ooo 


0t000 


0,000 


Large GrouP <>8) 


(* 


♦ 


1 


0 


0 


0 


0 


♦ 1 ♦ 


0*006 


0,000 


0,000 


None 09 the above 


5 


♦ 
♦ 
♦ 


1 


0 


0 


0 


2 


♦ 3 ♦ 
♦♦♦♦♦♦♦♦♦♦♦ 


,o67 


• 667 


,643 


Margi nal. 
> 


Total s 


♦ 

♦ 


28 


0 


0 


0 


2 


♦ 30 ♦ 









♦♦♦♦♦♦♦♦ 



♦♦♦♦♦♦♦♦ 



Category Marginal Agreement! *305 
J J ] Cohan ft Kappa) ,459 5 J 



ERiC 



Table 78 

Phase XV - IGE Evaluation Project 
Descriptive Study - Spring 1978 

Reliability on orouPlno 
Observer 30 Expert 80 
School 601 Grade 5 
Week o Day 5 

DMP 



Category chosen 
by. expert SO 



Category chosen by Observer 30 



Cot ■ > 
* 



1 



Mara I na I 
Totals 



Nominal 
Agreement 



Marginal 
Agreement 



Light's 
Extens Ion 
of Kappa 



Individual 




1 


♦ 


9 


0 


0 


0 


0 


♦ 


9 


♦ 1*000 


,818 


1,000 


Pair 




2 


♦ 
♦ 


I 


0 


0 


0 


0 




1 


♦ * o,ooo 


0,000 


0,000 


Snta 1 i Group 


(3*8) 


3 




1 


0 


0 


0 


0 


» 


1 


♦ o«ooo 


0,000 


0,000 


Large Group 


l>8) 




♦ 
♦ 


0 


0 


0 


12 


0 




12 


♦ ' 1.000 


1,000 


J ,000 


None of the 


above 


5 


♦ 


C 


0 


0 


0 


?■ 


♦ 


7 


♦ 1*000 


1,000 


1,000 



♦ 

Marginal Totals ♦ 11 0 

♦ 
♦ 



12 



+ 



30 



category Marginal Agraamenti »564 
Cohen's Kappa* ,901 



!G7 



9 

ERIC 



1 $8 



Table 79 



Phase iv * IGE Evaluation Project 
Oeicrlrclve Study - Spring 1978 

rgi labl I | ty on Grouping 
Observer *o Expert 80 
School 801 Grade 5 
Week 0 Day 5 
DMP 



ho 

00 



Category chosen 
by Expert 80 

/ 

Individual 
Pal r 

Sma I | Group (3~8) 
Large Group (>8) 
None of the above 



Category chosen by Observer 40 Light's 
Cat* Marginal Nominal Marginal Extension 

# l 2 3 4 5 Totals Agreement Agreement of KaPPa 



♦ 

1 ♦ 

2 ♦ 
♦ 

3 ♦ 
♦ 

♦ 

5 ♦ 



Marginal Totals ♦ 



9 
1 
0 
0 
0 

10 



0 ' 

0 

0 

0 

0 



13 



♦ 



30 



♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 



0 


0 


0 


♦ 


9 < 


> loOOO 


• 900 


1*000 


0 


0 


0 


♦ 
♦ 


I * 


► 0*000 


0*000 


0 t 000 


1 


0 


0 


♦ 
♦ 


1 4 


► 1*000 


1*000 


1*000 




12 


0 


♦ 
♦ 


12 4 


> 1*000 


• 923 


1*000 


0 


1 


6 


♦ 


7 < 


• 857 


• 857 


• 821 



1S«J 



Category Marginal Agreement I «736 
Cohents Kappa i *902 



170 



9 

ERIC 



Table 80 

PKatt XV - IOC Evaluation Project 
Descriptive Study - Spring 1978 

Reliability on Grouping 
Observer 10 Expert 90 
School 001 Grade 2 

Week 0 Day 5 -S) 
WDRSD 



Category cKosen 
by Expert 90 



Category choten by Observer 10 
Cat, Marginal 
* 12 3 4 5 Totali 



Nominal 
Agreement 



Marg| na| 
Agreement 



llgbt't 
Extens Ion 
o* Kappa 



Xndl vl dual 
Pair 

Smai | Group (3-8) 
large Group OB) 
None of the above 



1 ♦ 0 0 0 0 1 ♦ l 

2 ♦ 0 0 0 0 0 ♦ 0 

3 ♦ 0 0 0 0 0 ♦ 0 

♦ ♦ 

4 * 0 0 0 51 1 ♦ 52 

♦ ♦ 
5* 0 0 0 0 lv 1 

♦ ♦ 

Marginal Totals ♦ 0 o 0 51/3* 54 



0.000 



• <>81 
1*000 



0,000 



• 981 

• 333 



0,000 



• 654 
1,000 



Category Marginal Agreementi ,438 
CoKents Kappai ,586 



171 



J 72 



Table 81 

PKata IV - XGE Evaluation projact 
OatcrlPtlva Study - Spring 1978 

Re I |abl I I ty on Grouping 
Observer 20 Expert 90 
School 601 Grace 2 
Week 0 Day 5 
WDRSO 



O 



Category chosen 
by Expert 90 



Category chosen by Observer 20 



Cat, 



1 



5 



♦♦♦♦♦♦♦♦ 



\ 



Individual 


1 ♦ 


0 


0 


0 


0 


1 


♦ 1 ♦ 


o.ooo 


Pair 


2 ♦ 


0. 


0 


0 


0 


0 


♦ 0 ' ♦ 




Sma| I Group (3*8) 


3 ♦ 


0 


0 


0 


0 


0 


♦ o ♦ 




Large Group (>8) 


4 ♦ 


1 


0 


o , 


49 


2 


♦ 32 ♦ 


#942 


None of the above 


5 ♦ 


0 


0 


0 


1 


0 


♦ 1 ♦ 
♦♦♦♦♦♦♦♦♦♦♦ 


o»ooo 


Marginal 


Totals ♦ 


1 


0 


0 


50 


3 


♦ 54 ♦ 





Marg | na I 
Totals 



Nom I na I 
Agreement 



Lloht»s s 
Marginal Extension 
Agreement of Kappa 



1,000 




••019 



,221 
,059 



173 



Category Marginal Agreementi e T65 
Cohen's Kappal «13S , 



Table 82 

Phase IV - jot evaluation Project 
OetcrlPtlva Study - Spring 1978 

Re I | ob I I | ty on Grouping 
Observer 30 Expert ac 
School 801 Grade 2 
Week o Day 5 
WORSD 



Category chosen 
by expert 80 



Individual 
Pair 

Small Group <3-8) 
large Group (>8) 
None of the above 



Category chosen by Observer 30 
Ca l 9 , Marginal 
# 12 3 4 5 Totals 



♦ 



1 ♦ 

2 ♦ 

3 ♦ 
♦ 

4 ♦ 
♦ 

5 ♦ 



39 



Marginal Totals ♦ *3 



1 



o 

0 



1 

1 



♦♦♦♦♦♦♦♦ 

41 

6 



♦♦♦♦♦♦♦♦ 



Noml nal 
Agreement 



t951 

• 333 

o»ooo 

• 600 



Marginal 
Agreement 



• 953 

•soo 

• 500 

• 714 



Llghtts 
Extent I on 
of Kappa 



.761 
,294 



••019 
♦ 5*0 



9 



175 



Category Marginal Agreement! ,667 
Cohents Kappa* ,508 



J 76 



Table 83 ' 

Phase !v - IGE Evaluation Project 
Descriptive "Study - Sprlno 1978 

Re I labl I I ty on Grouping 
Observer 40 Expert 80 
School 801 , Grade Z 
Week 0 , Day 5 
' WDRSO 



Category chosen by Observer 40 



Category choten 
by Expert 80 



Individual 
Pa I r 

Sma| | Group (3«-B) 

Large Group <>8) 

None of the ab^ve 



Marginal Totals ♦ 




Marginal Nominal 
Totalt Agreement 



,976 
,833 



1,000 
1,000 



Light! $ - 
Margfnei Extension 
rgreement 6* Kappa 



1,000 
1,000 

1,000 
1,000 



,899 
,813 

1,00-0 
1,000 



category Marginal Agreamenti 1,000 J 7 J 

Cohen»s Kappai .908 

177 



ERIC 



Table 84 

Phase IV - IGE Evaluation Project 
Descriptive Study • Spring 1978 

Reliability on Grouping 

Observer 10 Expert 80 

School 801 Grade 2 

week o Day 5 

DMP 



Category chosen Cat. 
by Expert 80 H 



Category chosen by Observer 10 



Mar g | ne I 
Total s 



Nom I na l 
Agreement 



Mb ginai 
Agreement 



Light 's 
Extens I on 
of Kappa 



Individual 1 

Pair 2 

Smal | QrouP (3-8) 3 

Large Group <>8) k 

None of the above 5 



Marginal Totals 



30 
0 
0 
0 
0 



0 

1 

0 
0 

0 



0 
0 
k 
0 



1 

0 
0 
32 



0 
0 
0 
5 



♦ 

* 
♦ 

♦ 

♦ 

♦ 



31 
1 
<» 

37 




,968 
1 .000 
1 .000 
.865 
• 600 



,96* 
1.000 
1,000 
.946 
.625 



,9<#8 
1,000 
1,000 
,755 
,55<* 



Category Marginal Agreementl ,908 
Cohen's Kappa) ,836 



17!) 



9 

ERIC 



Mill 



Table 85 

Phase Iv - I6E Evaluation Projact 
Descriptive Study • Spring 1978 

Re I labl | |ty on Grouping 
Observer 20 Expert 80 
School 801 Grade 2 
week 0 Day 5 
DMP 



H 



Category chosen by Observer 20 



Category chosen 
by Expert 80 



Individual 
Pal r 

Sma I | Group (3-B) 
Large Group OS) 
None of the above 



Cat , 
u 



* 

1 «■ 
♦ 

2 ♦ 
+ 

3 ♦ 

4 ♦ 

5 ♦ 
♦ 



30 
1 

0 

0 
0 



I 

0 
0 
0 
0 



c 

0 

4 

0 
0 



0 
0 
0 
35 
3 




0 
0 
0 
2 
2 



Marginal Nominal 
Totals Agreement 



♦♦♦♦♦♦♦ 



♦♦♦♦♦♦♦♦♦ 
31 



1 

4 
37 
5 



♦♦♦♦♦♦♦♦♦ 
76 ♦ 
♦♦♦♦♦♦♦♦♦ 



• 968 
0 .000 
1.000 
,946 
• 400 



Marginal 
Agreement 



1.000 
1.000 
1,000 
,974 
t 800 



LlQht'S 
Extens I on 
of Kapoa 



,946 
«,013 
1,000 
,695 
,366 



1 1 



Category Marginal Agreement! <**5 
Cohen's Kappal ,652 



I v> 



9 

ERIC 



Table 86 

Phase IV - IGE Evaluation Project 
Descriptive Study - Spring 1578 



Re I I eb I I I ty 
Observer 30 
School 801 
Week 0 



Off Grouping 
^Expert 90 
Grade 2 
Day 5 
DMP 



Category chosen by Observer, 30 



Category chosen Cat, 
by Expert 90 * 



Individual 1 

Pair 2 

Sma I | GrouP (3~8) 3 

Large Group t>8> 4 

None of the above 5 



♦ 
♦ 
♦ 
+ 
♦ 
♦ 
♦ 

* 



Marginal Totals ♦ 



40 
16 
7 
0 
0 

63 



1 
7 

2 
0 
0 

lo 



0 

0 
0 
0 
0 



0 
0 
0 
0 
0 



0 
0 
0 
3 
8 

11 



Marg I na 
Total s 



♦ 

♦ 

♦ 

♦ 
♦ 
♦ 

♦ 



41 
23 
9 
3 
fl 

84 



• ♦ 
♦ 

♦ 
♦ 
♦ 

4 
♦ 

♦ 
♦ 
♦ 



Norn I na I 
Agreement 



♦ 976 

• 304 
0«000 
0.000 
1*000 



Light's 
Marginal Extension 
Agreement of Kappa 



,651 
,435 
0,000 
0,000 
,727 



,902 
,210 
0,000 
0,000 
1,000 



♦♦♦♦♦♦♦♦♦♦ 



9 

ERIC 



Category Marginal Agreement! ,363 
Cohen's Kappal ,414 



Table 88 

Phase IV • 1GE Evaluation Project 
Descriptive Study • Spring. 1978 

Rei labl I I ty on Grouping 
Observer <**0 Expert 90 
School 801 Grade 2 
Week o Day 5 

. DMP 



Category chosen 
by Expert 90 



Individual 
Pal r 

Smal j Group (3-8) 
Large Group (>«) 
None of the above 



Category chosen by Observer 10 
Cat, Marginal 
* 1 2 3 4 5 Totals 



Marginal Total 



62 



Norn | na I 
Agreement 





♦ 




















1 


♦ 


40 


0 


1 


0 


0 




41 


♦ 


,976 




♦ 












♦ 




♦ 




2 




15 


8 


0 


0 


0 




23 


♦ 


,348 




♦ 












♦ 




♦ 




3 


+ 


7 


1 


1 


0 


0 




9 


♦ 


• 111 




♦ 
















♦ 




4 


♦ 


0 


0 


0 


2 


1 


* 


3 


♦ 


.667 




♦ 












♦ 




♦ 




5 


♦ 


0 


0 


0 


0 


6 


♦ 


8 


♦ 


1.000 




♦ 
















♦ 





♦ 



+ 
* 



84 



♦ 
♦ 

* 



Marg | na | 
Agreement 



.661 
.391 
.222* 
,667 
,t 89 



I Igbt'S 
Extens I on 
of KaPPa 



,907 
,270 
,089 
,659 
1,000 



category Marginal Agreement! ,566 
Cohen»s Kappal ,501 



ERiC 



Table 87 



phase IV - IfiE Evaluation Project 
Descriptive Study - Spring 1978 

Reliability on Mater I a I s 
Observer 10 Expert 9o 
School 801 Grade 2 
week 0 Oay 4 
WORSO 



Material I Paper/pencil 



Category chosen 
by Expert 90 



used 

Not used 
Other 



Cat • 
u 



Category chosen 
by ^Observer 10 

1 2 3 

♦ 



1 ♦ 
♦ 

2 «• 

3 ♦ 
♦ 



Mara | na I Totals 



♦ 
♦ 



39 
3 
0 

42 



0 
12 
0 

12 



0 
0 
0 



Mara I na I 
Tota I s 

♦♦♦♦♦♦♦♦♦♦♦ 



♦ 
♦ 
♦ 
♦ 
♦ 

♦ 

♦ 



39 
15 
0 

54 



♦ 

♦ 
♦ 
♦ 

♦ 
♦ 



Nom | na I 
Agreement 



1,000 
.800 



Maro I na I 
Agreement 



,929 

• aoo 



i igHt »* 

Extens l on 
of Kappa 



i.ooo 
,743 



♦♦♦♦♦♦♦♦♦♦♦ 



Category Marginal Agreement! .864 
Cohen's Kappal .853 



; 



Table 89 



\ 



Phase IV « I6E Evaluation Project 
Descriptive study - spring 1978 

Re Mobility on Mater I a I s 
Ohserver\ Co Expert 9o 
School 801 Grade 2 
Week 0 Day 4 
WOfiSO 



Material i Paper/pencil 



Category chosen 
by Observer 20 



Category chosen 
by Expert 90 



Used 

Not used 
Other 



Cat • 



1 
2 
3 



Marginal Totals 



♦ 
♦ 
♦ 

♦ 
♦ 
♦ 

♦ 
♦ 



1 

37 
8 
0 

45 



Mar g I na 
Total s 



I 
5 

0 



1 
2 
0 



♦ 
♦ 
♦ 
♦ 

♦ 
♦ 
♦ 



39 
15 

0 

54 



♦ 
♦ 
♦ 

+ 

♦ 

♦ 
♦ 



Norn | na I 
Agreement 



.949 
• 333 



Merg I na I 
Agreement 



• 867 

• 400 
0*000 



Light's 
Extens I on 
of Kappa 



• 692 

• 250 



Category Marginal Agreementl ,422 



Cohen's Kappai .395 



Table ?0 



Phase IV » IGE Evaluation Project 
Descriptive. Study - spring 1978 

Re I | ab I I I ty on Mater I a I s 
Observer 30 Expert 80 
School 801 Grade 2 
Week 0 Day* k 
WDRSO 



S 



Material! Paper/pencl 



Category chosen 
by Expert 80 



Cat • 



Category chosen 
by Observer 30 

1 2 3 



Marg I na 
Totals 



Nom| na I 
Agreement 



Marg I na I 
Agreement 



L 1 ght • 9 
Extens l on 
of Kappa 

























used 


1 






2 


1 


♦ 


49 


♦ 


.939 


.980 






















0.000 


Not used 


2 


♦ 


0 


0 


0 


♦ 


0 










♦ 








• ♦ 




♦ 






Other 


3 


♦ 


U 


0 


7 




11 




• 636 


.727 














* 




♦ 







• 633 



• 580 



Marg I na I Tota I $ 



♦ 



50 



8 



♦ 



60 



♦ 
♦ 



1 \) 1 



Category Marginal Agreement! .569 
Cohen's Kappal ,605 



1!)2 



Table 91 

# 

Phase IV- IQE'Eval uat Ion Project 
Descriptive Study - Spring, 1978 



Rel lab l 11 tV ori: Mater la I s 
' observer 40 Expert SO 
S,chool G01 . Grade 2 
' We.ek 0 • Day 4 

jwDRSO j 



Material I Paper/pencil 



4> 
C 



Category chosen 
by Expert 80 



Cateqory chosen^ 
by Observer 40 



Cat. 

H 



Mara I na I 
Total s, 




Norn i na I 
Agreement 



.959 



.727 



Marg I na I 
Agreement 



,930 

0.00.$ 
.618 



I 



A 

. U I flbt ' S 
Extens I on 
of Kappa 



.755 



,679 



!93 



Category Marginal Agreement! 
Cohen's Kappaj ,715 



,599 



ERIC 



Table 92 



Phase IV - IGE Evaluation Project 
Descriptive Study - spring 1978 

Rellablllvy on Materials 
Observer lO Expert Bo 
School 801 Grade 2 
Week 0 Day 4 
DMP 



Material l Paper/pencil 



Category chosen 
*by Expert 80 



Cat, 



Category chosen 
by Observer 40 

1 2~ 3 



Marg I na I 
Totals 



NOm \ na I 



Marg I na I 



■Agreement Agreement 



Llght»s 
Extens | on 
of Kappa 















♦ 












Used 


1 




57 


2 


2 




61 


♦ 


#934 


,967 


',780 






♦ 








♦ 




♦ 








Not used * 


2 




2 


2 


0 


♦ 


4 


♦ 


.500 


,667 


,'462 






% 








♦ 




♦ 








Otjjer 


3 




0 


2 


17 


- + 


19 




..895 


1,000 


,864 




♦ 








♦ 
























♦ 












Marg I na i 


7ota 1 s 




59 


6 


19 




84 


♦ 


H 


















♦ 




♦ 









Category Marginal Afreementi 
Cohen's Kappal ,781 



878 



35 



196 



1 



Table 93 



Phase iy - I GE Evaluation Project 
Descriptive Study - SPrlnQ 1978 

f 

ReHabH-ttV on Materials 
Observer 20 Expert 80 
Schoo I 801 Grade 2 
Week 0 Day'V 
DMP 



\ 



H 
ho 



Category chosen 
», by Expert 80 



Cat, 
ft 



Mater I a I J Paper/penc I I 



Category chosen 
by Observer 20 



1 



Marglna 
Tota I s 



Nom | na I 
Agreement 



Mara I na I 
Agreement 



Light's 
Extens I on 
of Kappa' 







♦ 
















used 


1 


4 


59 


1 


1 




61 




, ,967 


» 




♦ 












♦ 




"Not used 


2 




i 


3 


0 




4 




• 750 






+ 








♦ 








Other 


3 




>i 


0 


18 


♦ 


19 




• 947 














♦ 




♦ 






































♦ 




Mar'g 1 na 1 


Tota 1 s 




61 . 


4 


19 


> ♦ 


84 



























1,000 #88a 
1,000 .737 
1, 000 .932 



Category Marginal Agreementl 1,000 
Cohen's Kappal ,886 



193 



ERIC 



N TabU 94 , . 

-phase fV « XOE Evetuat Ion Project' 
Descr/iPttvis Study . • spr I ng 1978 



ReH I nb I I I ty on Mater I a I s 
Observer 30 '„ Expert 90 
School 801 drade 5 
Week 0 Day 4 
s DMP 



Material I Paper/pencil 



Category chosen 
by Expert 90 



Cat, 

n 



Category chose/i 
by c Observer 30 



1 



.3 



Marginal Nominal Marginal 
Totals Agreement*"*, Agreement 



L I ght ♦ s 
Extens I on 
of Kappa 







<*• 


















. 1 


used 


1 


♦ 


54 


0 


1 




55 




.982 


• 902 


,917 






♦ 




* 
















Not used 




♦ 


5 


7 


3 




15 




V.467 


.667 


.388 


























Other, 


3 


♦ •* 


2 


3 


' 3 




e 


K 


,375 


.875' 


* .313 




♦ 






* 















Marg I na I Tota is* 61 



10 



78 



♦ 
♦ 



/ 



Category Marginal Agreements .814 
Cohen's Kappai ,S67 



ERIC 



4> 



?00 



ERIC 



Table 95 

Phase IV m IGE Evaluation Project 
Descriptive Study - SPr|no 1978 

Ro I | ab I I I ty . on Mater I a I s 
Observer 40 Expert 9o 
School 801 Grade 5 



Week 



Day 4 



OMP 



Mater | a I l , Paper /penc I I 



X 



Category chosen 
by Expert 90 



Cat, 



Category chosen 
by Observer 40 



1 



?. 



Marginal Nomina 



Marg l na I 



L I Qht » S 
Extens | on 



Totals Agreement Agreement o* Kappa' 















♦ 








Used 


1 




55 


0 


0 
y 


♦ 


55 


♦ 


1*000 


Not used 


2 


♦ 


5 


10 


0 




! 15 


♦ 


• 667 


















♦ 


.8/5 


Other 


3 


♦ 


1 


0 


7 


♦ 


8 


♦ 






♦ 








♦ 









Marginal 



Totals ♦ 



61 



10 



78 



♦ 



• ^ 0c 



667 



875 



ItOOO 
.6^8 
,863 



\ 



(Category Marginal Agreementi 
Cohen's KappaJ ,815 • 



1* 

• 814 



2j2 



. . t Table 96 

Phase IV - IflE Evaluation Project 
Descriptive -Study - spring 1978 

Re I I ab I I I ty on Mater I a I s 

■ Observer 10 Expert 90 

~> School 801 Grade 5, ,. \ , 

Week 0 , Day 5 



\ 



0MP 



r 



Mister I, a |-r^PaPer /pej>c I I 



Category chosen 
by Expert 90 



\ 



used r 
Not used 
Other 



.Cat • 
ft 



J 

Cv-»teaory chosen 
by Observer 10 



1 

2 ♦ 

•"N 

♦ 

3 ♦ 



1 



25 



Marginal Nominal Marginal 
Tota I 9 agreement Agreement 



0 
0 
0 



2 
0 
3 



Marginal Totals ♦ 



25 



♦ 



27 , ♦ 



0 
3 

30 



♦ f 



,9*26 



1.000 



/ 



203 



category Marginal fcoreementl .763 
Cohen's Kappal #71<e, 

V. 



.926 



.600 



1 1 ght 1 s 
Extens I on 
of Kappa 



,556 



1 ,0-00 



204. 



* Table 97 , 

Phase IV - JGE Evaluation Project 
Descriptive Study - spring 1978' 

Reliability on Materials 
Observer 20 Expert 9o 
School 801 Grade 5 
■ Week 0 Day 5 
DMP 



H 



Material t Paper/pencil 



Catecory chosen 
by Expert 90 



\ 



- Cat, 



Category chosen 
by Observer 20 



Maro I na I 
Toters 



Ndm | na I 
Agreement 



Marginal 
Agreement 



. Llght» t 
Extens | on 
of Kappa 





* 


♦ 




















Used 


1 


♦ 


27 ° 0 


, 0 




87 




It 000 


1 


.000 , 


1.000 






♦ 






V 














Not used 


2 




0 0 


0 




0 






0 


• 000 








♦ 




















Other 


3 


♦* 


0 1 


2 




3 




• 667 




♦ 667 


,6A3 






♦ 


































































Maro 1 na 1 


Tota 1 s 




27 1 


2 




30 
















♦ 





















category Marginal Agreementl .556 
Cohen's Kappal »818 



20G 



9 

ERIC 



Table 98 

Phase IV - IGE Evaluation Project 
Descriptive Study -Spring 1978 

Rcl Iflbl I 1 1 y or. yater lals 
Observer 30 Expert 80 
School 801 Graae 5 
week 0' Day 5 

DMP ^ 



Materiel! PaPer/pene I I, 



4 



Category chosen 
by Expert 80 



Cat* 



Ceteoory choserS 
by ^bftpryer 30 



1 



Marg I na I 
Tota I s 



Norn | na I 
Agreement 



Mars I ha l 
Agreement 



LlQhtt*. 

Extens l on 
of Kappa 



Used 

Not used 
Other 



826 



♦ . \ ♦ 

1 ♦ 19 4 0 ♦ 23 ♦ 

♦ ' ♦ ♦ 

2 ♦ 0 0 0 + 0 ♦ • 

3 * 0 0 7 7 ♦ *'1#000 

♦ ♦' ♦ 



• 826 
0*000 
1*000 



,526 



1*000 



, ♦ 

'Marginal Totals ♦ 

♦ 



19 



♦ 



30 



♦ 



Category Marginal Agreementl »609 
Cohen* s Keppal ,710 ' 



207 



08 



6 



Table 99 



Phase IV - IGE Evaluation Project 
Descrietlve Study - spring 1978 

Reliability on Materials 
Observer 40 v Expert 80 
School 801 Grade 5 
Week 0 ' Day 5, 
DMP 



Material! Paper/pencil 



00 




Category chosen 
by Expert 80 



Cat, 
* 



Cateaory chosen 
by Observer 40 



Mara I na 
Totals 



Nom| na I 
Agreement 



Marginal 
Agreement 



Llght»s 
Extens l on 
of Kappa 



Used 

Not used 
Other 



Marg I na I 



♦ 

Y ♦ 
♦ 

2 ♦ 
♦ 

3 ♦ 
♦ 



Totals ♦ 



23 


0 


0 


0 


0 




1 


0 


6 



24 



♦ 

♦ 



23 
0 
7 

30 



♦ 
♦ 



1 tOOO 



f 958 



;857 #857 



uobo 



821 



9M'J 



• I 

category Marginal Agreementl »908 
Cohen's Kappa i %902 



2: o 



o 

ERIC 



* 



Table LOO 

Phase"**IV • I6E Eva I uat I nn Project 
Descriptive Study - Spring i978 . 

Re I lab | l | I ty on Mater I a I s 
Observer 10 Export 90 
School 801 Grade 2 
Week 0 Day 5 
WDRSD 



Category chosen 
by Expert 90 



Used 

Not used 
Other 



Cat 



♦ 

1 + 

2 ♦ 
♦ 

3 ♦ 
♦ 



Material J PaPer/penci 



Cateaory chosen 
by Observer 10 



1 


2 


3 


0 


6 


0 


0 




2 


0 


0 


1 



parQ i ne I Nom | na I 
Tota I s Agreement 



♦ 



6 
47 
I 



♦ 



0.060 
• 957 
1.000 



Marg l nA I 
Agreement 



0.000 
.922 
.333 



Light «s 
Extens | on 
of Kappa 



0,000 
,231* 
1,000 ' 



Marginal Totals ♦ 

♦ 



51 



54 



♦ 
♦ 



category Marginal Agreementl ,418 
Cohen's Kappas ,163 



?'2 



Table 101 

Phase IV - IGE Evaluation Project 
Descr | pt I v*» Study -SPr lnO 1978 

• Reliability on Materials* 
Observer' 20 Expert 90 ' 
School 801 Grade 2 
Week 0 Day 5 
'* WDRSD 



Material I Paper/pencil 



Category chosen 
by Expert 90 



Used 

Not used 
.• other 

Mars I na I 



Cat, 



Category chosen 
by Observer 2o 



Marginal ' Nominal 
Totals Agreement 



Marg I na I 
Agreement 





♦ 


















1 


♦ 


0 


6 


0 


♦ 


6 


♦ 


0.000 


0.000 
















♦ 




.922 


2 


♦ 


'0 


44 


3 




47 




#936 




















.333 


3 




0 


I 


0 


♦ 


1 




0t000 




» ▲ x ▲ 






























i 








s 


♦ 


0 


51 


3 


♦ 


54 










♦ 



















Lightts 
Extens | on 
of Kappa 



0,000 
• •149 



• 059 



Category Marginal Agreement! ,<»1B 
Cohen's Kappa; -,047 



214 



Table 102 « 

Phase iv - Evaluation Project 
Descriptive Study - spring 1978 



ReJ l ab f I I ty on Mater I a I s 
Observer 30 Expert 80 
School 801 Grade 2 
leek 0 Day 5 
WDRSD 



later | a I l Pe'per/penc I I 



Category ch-osen 
by Expert eft 

• ( 

Used 

Not used 
Other 



Cat « 



CAteqory chosen ,p 
• by Observer 30 



/ I 



7 



Marginal Totals ♦ 



Marginal Nominal Marginal 



Lioht is 
Extens |on 



Totals Agreement Agreement of Kappa 







• 1 








if 


Ml 




. 3 


♦ 


47 


i ♦ 








* 






1 


1 


0 


♦ 


2 


/ : 












12 * 


2 


0 • 


3 


♦ 


S 



, .872 
.500 




• 600 



.936 
• 500 



• 633 



• 311 
,460 

\ - 



9 

ERIC 



215 



Category Mar g I no I Agreement t #757 
. Cohen's Kappat .400 



CI 



s 



. , Table 1£>3 

Phase IV - I GE Evaluation Project 
Descriptive study - SPr I ng 1978 



Re I I ab l l 1 1 v on Mater l a I $ 
Observer 40 Expert 8a 
School 801 Grade 2 
Week 0 Oay 5 
- WDRSO » 



Material l Paper/pencil 



Category chosen 
by Expert 80 



used 

Not used 



Other 



Cat, 



1 



(Marginal Totals ♦ 



i 



tegory chosen 
by Observer 40 



1 



Marfll na 
Total s 



Norn | naj 
Agreement 





♦ 








* * 








1 


♦ 




0 


0 


♦ 


47 


♦ 


1*000 




4 








♦ 




♦ 






♦ 


2 • 


0 


0 


♦ 


a . 




0.000 




♦ 
















3 




0 


0 


5 • 


♦ 






I #000 





















5 



♦ 
♦ 
♦ 



54 



♦ 
♦ 



Marg I na I 
Agreement 



,959 
0^.000 
1,000 

i 



Light's v 
Extension 
of Kappa 



1,000 
0,000 
1,000 



217 



category Marginal Agreementl 
Cohen's Kappai ,816 



,653 



/ 



218 



ERIC 



# 1 



Table 104 



Phase iv - IGE Eva I vetr : sr» Project 
Descriptive Study - Spring 1978 

Re I 1 8bl I I ty on Mater I a I s 
Observer 10 Expert 80 
Schoo I 801 Grade 2 
Week 0 Day 5 
DMP 



r 



Material I Paper/pencil 



Category chosen 
by Expert 80 



Cat, 
n 



Cateqory chosen 
by Observer 10 



1 



Mapg I na I 
Tota I s 



Nom | na l 
Agreement 



Marginal 
Agreement 



LlQht»s 
Extens | on 
of Kappa 



Used 

Not used 
Other 



♦ 

1 + 
♦ 

2 ♦ 
♦ 

3 * 



44 
0 
1 



1 

23 
1 



0 
5 
3 



♦ 

♦ 

♦ 
♦ 



45 
28 
5 



♦ 
♦ 
♦ 
♦ 

♦ 



,978 

• 821 

• 600 



1 »000 

• 893 

• 625 



,947 
,737 
,554 



Marginal Totals ♦ 

♦ 



45 



25 



8 



♦ 
♦ 



78 



♦ 
♦ 
+ 



23.9 



Category Marginal Agreement! 
Cohen's KappM .81? 



,839 



220 



Table 105 



Phase IV • IGE Evaluation Project 
Descriptive Study - spring 1978^ 

Re I I ab I I I ty on Mater I a I s 
Observer 20 Expert 8o 
School 801 Grade 2 
Week 0 Day 5, 
DMP 



Material l Paper/pencil 



Category chosen 
by Expert 80 



Cat • 



Category chosen 
by Observer 20 



Maro I na I 
Tota I s 



Norn | na I 
Agreement 



Marginal 
Agreement 



. Llghtis 
Extens | on 
of Kappa 







♦ 












♦ 








used 


1 


♦ 


42 


3 


0 • 




45* 


♦ 


• 933 


.978 


,838 






♦ 




















Not used 


2 


♦ 


1 


25 


2 


♦ 


28 

* 


♦ 


• 893 


1.000 


' ,833 


























Other 


3 


♦ 


3 


0 


2 




5 


♦ 


• 400 


• 800 


• 368 






♦ 








+ 




♦ 












♦ 












♦ 








Marg 1 na 1 


Tota l s 


♦ 


46 


28 


4 




78 


♦ 

































♦♦♦♦♦ V 



Category Marginal Agreement! «926 
Cohen's Kappal «78l 



ooo 

rw <V 



Table 10b 

Phase IV - IGE Evaluation Project 
Descriptive Stddv ^spring 1978 

Re I I ab.l I I ty on Mater I a I s 
Observer 30 ' Expert 90 V 
School 801 Grade 2 
Meek 0 Day 5 
DMP 



J Material! PaPer/pencll 



Category chosen 
by Expert 90. 



v 



used 

Not uted 
other 



Cat « 
n 



1 ♦ 
♦ 

2 ♦ 

3 ♦ 
♦ 



Category chosen 
by Observer 30 



1 



Marginal Totals ♦ 

♦ 



73 
0 
0 

73 



0 
0 
0 



0 
3 
6 

n 



Marg I na I 
Tota I s 



♦ 
♦ 

♦ 
♦ 

♦ 



73 
3 

e 

84 



♦ 
♦ 

♦ 







Light's 


Nom | na 1 


Marg I na 1 


Extens Ion 


Agreement 


Agreement 


of Kappa 


1 .000 


1.000 


1.000 


0,000 


0.000 


0,000 


1.000 


.727 


1,000 



9 

ERIC 



nO'X 



category Marg I na I Agreementi .576 
Cohort's Kappal .846 



224 



Ui 



Table 107 

Phase IV - IGE EvafuatJ^ p-oject 
Descriptive Study - spring 197B» 

Rel labl I 1 1 v on Materials 
Observer 40 Expert 90 
School bOl Grade 2 
week 6 ' Day 5 
DMP 



Mater I a 1 1 Paper/penc I I 



Category chosen 
by Expert 90 



Cat* 



Cateaory chosen 
by Observer 10 



Marg I na I 
Tota I s 



Nom 1 na I 
Agreement 



Margin I 
Agreement 



Light's 
Extens | on 
of Kappa 



Used 

Not used 
Other 



Marginal 



♦ 

1 ♦ 
♦ 

2 ♦ 

3 ♦ 
♦ 



Totals ♦ 
♦ 



73 

0 
0 

73 



0 
2 
0 



0 
1 

a 

9 



♦ 
♦ 
♦ 

♦ 
♦ 
♦ 

+ 

+ 



73 

3 

e 

8* 



♦ 
♦ 
♦ 
♦ 
+ 
♦ 

♦ 

♦ 



1.000 
,667 
1,000 



1,000 
,667 
,889 



1.000 
,659 
1.000 



9 O ^ 



Category Marginal Agreementl .852 
Cohen's Kappel ,949 



(C \J 



9 

ERIC 



* Table. 108 

* 

Phase IV - IGE Evaluation Project 
Descriptive Study - Spring 1978 



Re I | ab I I Itv on Materials 
Observer 10 Expert 80 
School 801 " Grade 2 
Week 0 Day 4 
DMP 



Material I ManlPulatlv 



1 



Category chosen 
by Expert 80 



Cat. 
n 



Used v. 


l 


♦ 8 


1 


0 


Not used 


2 


♦ I 


S3 


2 


Other 


3 


♦ 0 

♦ ♦♦♦♦'♦♦<( 


2 


17 


Marg 1 na 1 


Total s 


♦ .9 


56 


19 



Category chosen 
by Observer .40 

1 2.3 
+ + + + + + + + + 



Marg I na I 
Total s 



♦ 
♦ 

♦ 



9 
56 
19 



84 



Nom | na I 
Agreement 



• 889 
.946 

• 895 



Llght«8 
Marginal Extension 
Agreement of Kappa 



1.000 
1.000 
1.000 



,676 
.839 
,864 



category Marginal Agreement! 1.000 
Cohen's Kappa i .855 



227 



228 



/ 

Table 109 

i 

Phase,. IV - IGE Evaluation Project 
Descriptive Study » spring 1978 

Re I I ab I I I ty on Mater I a I s 
Observer 20 Expert 80 
School 801 Grade 2 
week 0 Day 4 
DMP 



Mater ial t ManlPuiat I ve 



Category chosen 
by Expert 80 



Cat, 



Category chosen 
by Observer 20 



1 



X 

tfarg I na I 
Tota I s 



Nom | na I . 
Agreement 



Marg I na I 
Agreement 



Light's 
Extent | on 
of Kappa 



Used 

Not used 
Other 






♦ 


















1 


♦ 


6 


3 


0 


V 

♦ 


9 


* 


• 667 


• 667 
















♦ 








♦ 


0 


55 


1 




56 




• 982 


.949 






















3 


♦ 


0 


« 
1 


18 




19 






1.000 




♦ 












4 







.641 

.940 
.932 



Marginal Totals ♦ 



59 



19 



♦ 
♦ 
♦ 



84 



♦ 



Category Marginal Agreementl .872 



9 '! 



'30 



Cohen's Kappal .874 



Table 110 

Phase XV • XGE Evaluation Project 
Descriptive Study - sprlno 1978 

Re I l ob I I I ty on Material* 
Observer 30 Expert 90 
i School 801 Grade 
Meek 0 Day 4 
OMP 



/ 



/ 



s 



T 



Mater La I J Manipulative 



Category chosen 
by Expert 90 



Cat, 



Cateaory chosen 
by Observer 30 



1 



Marginal < / Nominal 
Totals /Agreement 



UlgHt«s 
Marginal Extension 
Agreement of Kappa 





♦ 
















• 767 


Used 


1 ♦ 


20 


3 


0 




23 




• 870 




♦ 












♦ 






Not used 


2 + 


10 


33 


4 




47 




• 7 02 


• 872 


■ 


♦ 


















Other • 


3 ♦ 


0 


5 


3 




e 


♦ 


• 375 


• 875 


i 






















♦ ■i > 




















♦ 








♦ 










Marginal 


Totals ♦ 


30 


41 


7 


4 


78 









,788 
,372 
,313 



ERIC 



231 



category Marginal Agreement! 
Cfohen»s Koppal ,497 



838 



/ 



\ 



H 



900 



mm 



ERIC 



Table 111 

Phase IV - IOE Evaluation Project 
Descriptive Study • spring 1970 

Re I | ab I I I ty on Material s 
Observer 40 Expert 90 
School 801 Grade ^ 
Week 0 Day 4 

DMP 



a 
o 



Material I Man | pu I at I ve 



Category chosen 
by Expert 90 



Used 

Not used 
Other 



Cat. 
u 



Category cnosen 
by Observer 40 



1 



Marg I na I 
Total s 



















1 




21 


2 


0 




23 




















2 




20 


27 


0 




47 




















3 


♦ 


0 


1 


7 




e 






4 








+ 







Nom | na I 
Agreement 



.913 
.574 
.675 



LlghtM 
Marginal Extension 
Agreement of Kappa 



.561 .817 
,638 ,309 
,875 .863 



Marginal Totals ♦ 41 



30 



♦ 



78 



♦ 



x.i 



Category Marginal Agreementl ,691 
CoKen's Keppal ,512 



V\ 1 
~ <> 'i 



Table 112 

Phase IV - IOE evaluation Project 
OeserlPtlve Study - SprlnO 1978 w 

•v 

Re I lability on Mater I a I $ 
Observer 10 Expert 90 
ScHool 001 , Grade 5 
Week 0 Day 5 
DMP 



\ 



Material I Manipulative 



Category chosen 
by Export 90 



Catt. 
H 



) 



Category chosen 
by Observer 10 

1 2 3 







♦ 








♦ 






Used 


1 


♦ 


9 




1 


♦ 




♦ 


Not used 


2 


♦ 

♦ 


0 


12 


1 


+ 


13 




Other 


3 




0 


0 


3 




3 


♦ 



Marginal Nominal 
Tota I s * Agreement 



,643 
,923 
1,000 



Llgnt'S 
Marainal Extension 
Agreement of Kappa 



Marg I na I 



Totals ♦ 



9 



16 



♦ 
♦ 
♦ 



30 



♦ 
♦ 



category Marginal Agreement! #665 
Cohen's Kappal ,673 



ooi' 

L> O O 



,643 
,813 
,600 



236 



,490 
,835 
1,000 



fable 113 

Pha$e !V • IGE Evaluation Project 
# Descriptive Study - Spring 197§ 

Re I lab I I 1 1 v on Material! 
Observer 2o Expert 9o 
School 601 Grade 5 
Week 6 Day 5 
OMR 



Material t Manipulative 



Category chosen 
by Expert 90 



Cat • 
n 



Cateaory chosen, 
by Observer 80 



a 



Mara I na I 
Total s 



Nom I na I 
Agreement 



Marginal 
Agreement 



Llghtis 
Extent | on 
of Kappa 







♦ 
















Used 


1 




13 


1 


0 




U 




• 929 




















Not u*ed 


2 




0 


13 


0 




13 




1.000 






♦ 








♦ 








other 


■ 3 


♦ 


0 


1 


2 


♦ 


3 




• 667 




♦ 
















* 




♦ 
















M^pfllnal 


Total S 




13 


15 ' 


2 




30 






















♦ 





.929 
.867 
.667 



,874 
1.000 
.643 



or? 



Category Marginal Agreement? .821 
Cohen's Keppel .884 



238 



9 

ERIC 



/ 



Table 114 

Phase IV - IflE Evaluation Project 
•Descriptive Study - Spring 1976 

Rel lab! I 1 1 V on Materials 
Observer 20 Expert 90 
School 801 Grade 2 
Week 0 Day 5 
WDRSD 



Material l Manipulative 



Category chosen 
by Expert 90 



Cat • 



Category chosen 
by Observer 20 



Marg I na I 
Total s 



Norn | na I 
Agreement, 



Marg I na I 
Agreement 



UOhtt* 
Extens I on 
of Kappa 



Used 

Not used 
Other 



1 
2 
3 



0 
50 
1 



Marginal Totals 



♦ 

♦ 
♦ 
♦ 

♦ 

♦ 51 
♦ 



0 
0 
0 



0 
3 
0 



53 



♦ 

♦ 
♦ 
♦ 
♦ 

* ♦ 

♦ 54 ♦ 



♦ 
♦ 



* / 1 



0,000 
0*000 



0,000 
0,000 
,333 



0,000 
• ,059 



230 



Category Marginal Agreement! 
Cohen's Kappat -,001 



.111 



210 



Table 115 



\ 



Phase IV - I6E Evaluation Project 
DescrlDtlvc Study • Sprlr.3 1978 

Reliability on Mater I a I $ 
Observer 10 , Expert 80 
School 801 Grade 2 
week 0 Day 5 
DMP 



a- 



Mater |al I Man I pu I at I ve 



Category chosen 
by Expert 80 



Cat, 



Cateaory chosen 
by Observer 10 



Mara I na I 
Tote I s 



Now) na I 
Agreement 



Merg I na | 



L I Qht » s 
Extens i on 



Agreement of Kappa 







♦ 
















.920 


used 


1 


♦ 


80 


3 


0 


♦ 


23 


♦ 


• 870 














♦ 




♦ 






Not used 


2 


♦ 


5 


40 


5 




50 




• 800 


• 900 














♦ 




♦ 




• 625 


Other 


3 


♦ 


0 


2 


3 




5 


4 


• 600 






♦ 








♦ 




♦ 







,808 
• 527 
,554 



Marginal Totals ♦ 25 

♦ 



45 



8 



♦ 
♦ 
♦ 



70 



♦ 



:. x L 



category Marginal Agreement; »8l5 
Cohen's Kappal t 637 



9 A o 



9 

ERIC 



Table 116 



Phase IV - K?E Fvaluatlon Project 
Descriptive Study - Spring 197-8 

Re l | ab I I I t v on Mater I a I s 
Ohserver 20 Expert Bo 
School 801* Grade 2 
Week 0 Day 5 
DMP 



Mater | a I : Man l pu | at I ve 



Category chosen 
by Expert 80 



Cat , 



Cateaory cKosen 
by Observer 20 



1 



Marg| na 
Tota I s 



Nom | na I 
Agreement 



Mara I na I 
Agreement 



Ught»s 
Extens I on 
of Kappa 







♦ ■ 








♦ 












Used 


1 


♦ 


22 


1 


0 


f 


23 




• 957 


• 621 


• 932 






♦ 




















Not used 


2 


♦ 


6 




2 




50 




• 840 


• 920 


• 610 


























Other 


3 


♦ 


0 


3 


2 




S 




• 400 


• 800 


• 368 






♦ 





















Marginal Totals ♦ 



28 



46 



78 




Category Marginal Agreement! f 847 
Cohen 1 ! Kappal f 700 



243 



244 



•I 



Table 117 L 

Phase IV - IGE Evaluation Project 
Descriptive Study - Spring 1978 

Re I | ab I I I ty on Mater I a I s 
Observer 30 Expert 90 
School 801 Grade 2 
Week 0 Oay 5 
• DMP 



a 



Materi al t Manipuiat ive 



Category chosen 
by Expert 90 



Cat » 



Category chosen 
by' Observer 30 



Merg l na 
Tota I s 



Norn | na I 
Agreement 



Mara I na I 
Agreement 



Llghtts 
Extens | on 
of Kappa 







♦ 




















used 


1 




Ml 


8 


0 




49 


4 


♦ 837 


• 878 


,666 






















> 


,712 


Not used 


2 


♦ 
♦ 


2 


22 

• 


3 

i 




27 




• 815 


.900 


Other 


.3 


♦ 


0 


0 


8 




6 




1 tOGO 


#727 


1,000 






♦ 
























♦ ♦ ♦ 4 ♦ ♦ ♦ -f ♦ ♦ r ♦ 
















r. 




♦ 












♦ 






Marginal 


Totals 


♦ 
♦ 


A3 


30 


11 


* 


84 











9 A r i 



Category Marginal Agreement! ,835 
Cohen's Kappal ,730 



9 t 



o 

ERIC 



Table 118 



Phase IV - K>E Evaluation Project 
Descriptive Study - soring 1978 

Reliability on f^ater I a I s 
Observer <frO Expert 90 
School B01 Grade 2 
week 0 Day 5 
DMP 



Material! Manipulative 



Cateqory chosen 
by Observer 10 



Category chosen 
by Expert 90 



Used 

Not used 
Other 



Cat, 



♦ 

1 ♦ 
♦ 

2 ♦ 

3 ♦ 
♦ 



Marginal Totals ♦ 

♦ 



<»1 
3 
0 

44 



8 
23 

0 

31 



0 
1 
8 



Marg I na 
Tota I s 



♦ 
♦ 

♦ 
♦ 
♦ 



49 
27 
8 

84 



♦ 
♦ 
♦ 
♦ 

♦ 

♦ 



Nom I na I 
Agreement 



,837 
,853 
1,000 



Marginal 
Agreement 



,898 
,871 
.889 



Ugnt »s 
Extens l on 
of Kappa 



,657 
,765 
1,000 



category Marginal Agreementl 
Cohen's Kappal ,747 



,886 

248 



Table 119 



Phase TV - I*E Evaluation Project 
Descriptive Study - Spring 1978 

Re I I ftb I I I tv on Mater I a I s 
Observer 10 Expert 90 
School 601 Grade 2 
Week 0 Day 5 
WORSD 



a 

00 



Matpr I a It Game 



Category chosen 
by Expert 90 



Used 

Not used 
Other 



Cat, 
H 



Cateoory chosen 
by Observer lo 



Marg I na I 
Total s 



Marginal Totals 





♦ 








♦ 




- 4 


1 


♦ 


51 


0 


2 


4 


53 


4 




4 








4 




4 






0 


0 


0 


4 


0 


4 




4 








4 




4 


3 


♦ 


0 


0 


1 


4 


1 


4 




♦ 








4 




4 




4++++++++ 






















4 




4 


s 


♦ 


51 


0 


3 


4 


54 


4 




♦ 








4 




4 



NOm | na I 
Agreement 



• 962 



1*000 



Marginal 
Agreement 



,962 



• 333 



1 1 ght i s 
extens | oh 
of Kappa 



,321 



1,000 



category Marginal Agreement! «6M 9£(J 



u Cohen's Kappa I .486 



o 

ERIC 



Table 120 

Phase IV • I6E Evaluation Project 
Descriptive Study - Spring 1978 

Re I I ab I I I ty on Mater I a I s 
Observer 20 Expert 90 
School 801 Grade 2 
Week 0 Day 5 
WDRSD 



Material! Game 



Category chosen 
by Expert 90 



Category chosen 
by 0bserverv20 



Cat. 



Marg I na I 
Tota I s 



Nomj na I 
Agreement 



♦ ♦♦♦♦ 















4 








Used 


1 




0 


50 


3 




53 




0,000 


















+ 




Not used 


* ? 




0 


0 


0 




0 






















+ 




Other 


3 




0 


1 


0 




1 




0.000 























Light' 8 
Marginal Extension 
Agreement of Kappa 



0.000 
0.000 
.333 



0,000 



.059 



Mar g I na I 



Totals* 0 
+ 

++++++++ 



51 



5<» 



9 

ERIC 



251 



Category Marginal Agreement! • 1 1 1 
Cohen's Kappas ».001 



252 



Table 121 



Phase IV - XOE Evaluation Project 
Oescriotlve Study - spring 1978 



H 
O 



Re I I ab I I I ty on Mater l a I s 
Observer 30 Expert 90 
School 801 Grade 2 
Meek 0 Day 5 
DMP 



Material I Game 



Category cbosen 
by Expert 90 



Cat. 



Category chosen 
by Observer 30 



1 



Mara I na 
Tota I s 




Nom | na I 
Agreement 



0.000 
1 .000 
1.000 



Marginal „ 
Agreement 



Llghtts 
Extens I on 
n$ Kappa 



0.000 0»000 
1,000 1.000 
,727 1,000 



<> r. \ 



Category Marginal Agreementl ,576 
Coheres Kappal ,8*6 



2 r * 



•1 



9 

ERIC 



Table 122 



Phase IV - IGE Evaluation Project\ 
Descriptive Study - spring 1978 

Reliability on Materials 
Observer 40 Expert 90 
School 801 Grade 2 
week o Day 5 
DMP 



Mater I all Game 



Cateoory chosen 
by Observer 10 



Category chosen 
by Expert 90 



Used 

Not used 

Otfcer 



Cat • 
* 



♦ 

1 ♦ 
♦ 

2 ♦ 
♦ 

3 ♦ 



Marginal Totals ♦ 

♦ 

/ 



i 


2 


3 


0 


2 


( 

i 


0 


73 


0 


0 


0 


a 



75 



Mapg I na I 
Totals ( 



♦ 

♦ 
♦ 
♦ 
♦ 
♦ 

♦ 

♦ 



3 
73 

e 

84 



♦ 

♦ 
♦ 
♦ 

♦ 



Nom | na I 
Agreement 



0,000 
1.000 
1.000 



Marginal 
Agreement 



0.000 
.973 
.889 



Light* s 
Extens I on 
of Kappa 



0,000 
1 .000 
1.000 



255 



category Marginal Agreementl .621 
Cohen»s Kappal .833 



256 



Table 123 



Phase IV - IOE Evaluation Project 
Descriptive Study - SPMno 1978 

Re I 1 ab I I I ty on Mater I a I s 
Observer 30 Expert 90 
School G01 Grade 2 
Week 0 Day 5 
DMp 



N2 



Material) Audio/visual aid 



Category chosen 
by Expert 90 



Cat • 
u 



Category chosen 
by Observer 30 



Marg I na 
Tota I s 



Nom l na I 
Agreement 



Marginal 
Agreement 



Light »s 
Extens | on 
of Kappa 



used 

Not used 
Other 



1 

2 
3 



Marginal Totals 



♦ 
♦ 
♦ 
♦ 

♦ 

♦ 



0 

0 
0 



1 

72 
0 

73 



0 
3 
6 

11 



♦ 



1 

75 
8 



♦ 

♦ 
♦ 

♦ 



0.000 
• 960 
1*000 



0,000 0,000 
,973 #695 
,727 1,000 



Category Marginal Agreementi ,567 
Cohen's Kappal ,775 



C 5 O 



ERIC 



Table 124 

Phase IV - K'E Evaluation Project 
Descriptive Study - spring 1978 

Pel I ab I I I ty on Materials 
Observer 4-0 E^P^t 90 
School 801 Grace 2 
week 0 Day 5 
DMP 



Material! Audlo/vlsuel aid 



Category chosen 
by Expert 90^ 



Used 

Not used 
Other 

Marg I na I 



Category chosen 
by Observer 10 



Cat. 
H 

♦ 

1 ♦ 

2 ♦ 
♦ 

3 ♦ 

•f 

♦ 

Totals ♦ 



0 
0 

0 



1 

74 

0 
75 



Marginal 
Tota I s 



No* l na I 
Agreament 



0 

1 

8 



♦ 
♦ 

♦ 
♦ 



1 

'75 
8 

84 



♦ 
♦ 

♦ 



0 .000 
• 987 
1.000 



L I ght ♦ sv 
Marginal Extension 
Agreement of Kappa 



0.000 
1.000 
,889 



0,000 
,876 
1,000 



25;) 



category Marginal Agreement! .630 

» 

cohen»s Kappa? .876 



Table 125 



Phase IV - IGE Evaluation Project 
Descriptive Study - SPrlno 1978 



Re I I *b I I I ty on 
Observer 10 
School 801 
week 0 



Materials 
Expert 90 
Grade 2 
Day 4 



WDRSD 



Material! Printed material 



Category chosen 
by Expert 90 



Cat. 
u 



Category chosen 
by Observer in 



1 



MarO I na I 
Total s 



Nom | na I 
Agreement 



Marg | na I 
Agreement 



Lloht »s 
Extens l on 
of Kappa 



used 

Not used 
Other 



1 ♦ 

♦ 

2 ♦ 
♦ 

3 ♦ 



Marginal Totals ♦ 

♦ 



21 
2 
0 



27 

0 



23 31 



0 
0 
0 



♦ 
♦ 
♦ 
♦ 
♦ 

♦ 

♦ 

♦ 

♦ 



25 
29 
0 

54 



+ 
♦ 
♦ 
♦ 
♦ 
♦ 

♦ ♦ + 

♦ 
♦ 



.840 
• 931 



920 



<>935 



.721 



.838 



Category Marginal Agreementl ,92r 
Cohan's Kappal ,775 



'if ' > 



Table 126 , 

Phase IV - IGE Evaluation Project 
Descriptive Study - spring 1978 

• Reliability on Mater | a ! s 
Observer 20 Expert 90 
School 801 Grade 2 
Week 0 Day 4 
WORSD 



Material! Printed materia 



Category chosen 
by Expert 90 



Used 

Not used 
Other 



Cat, 
u 



1 
2 
3 



Marginal Totals 



♦ 
♦ 

♦ 
♦ 

* ■ 



Category chosen 
by Observer 20 



16 
6 
0 

22 



21 

0 

29 



Mere I na I 
Tota I $ 



Norn | na I 
Agreement 



Marginal 
Agreement 



/ 

Category Marginal Agreement! .627 
Cohen's Kappal #398 



L I Qht I S 

EXtens | on 
of Kappa 

















1 




25 






• 880 


• 392 
















2 




29 




o724 


ItOOO 


.404 
















0 




0 






0.000 
































3 





























in 



9 

ERIC 



'> r ) \\ 



2G4 



Table 127 

Phase $ - IGE Evaluation Project 
Descriptive S<.udy - spring 1978 

Re I I ab I I 1 1 y on rater I a I s 
Observer 30 Expert 80 
School 801 Grade ?. 
week o Day 4 



Material! printed material 



Category chosen 
by Expert 80 



Cat. 
H 



Cateaory chosen 
by Observer 30 
f 

1 2 3 



Maro I na I 
Total s 



Nom I na I 
Agreement 



Mara | na I 
Agreement 



1 1 ght ' $ 
Extens | on 
of Kappa 



Used 

Not used 
Other 





♦ 








4* 






1 


+ 


0 


0 


0 




0 




♦ 














2 


♦ 


... 4 


44 


1 




49 






♦ 








♦ 






3 


♦ 


0 


4 


7 




11 






♦ 












♦ 



Marginal Totals ♦ 

♦ 



48 



6 



♦ 
♦ 
♦ 



60 



♦ 
♦ 



.898 
.636 



0.000 
.980 
.727 



.490 
,580 



Category Marginal Agreement! 
Cohen's Kappal .534 



.569 



Table 128 

/ 

Phase IV - IOE Evaluation Project 
Descriptive Study - spring 1978 

Reliability on Mater I a I s 
Observer 40 Expert 80 
School 801 Oreae 2 
Meek 0 Day 4 
WQRSD 



Material! Printed material 



Category chosen 
by Expert 80 



Cat. 
u 



Category chosen 
by Observer 40 



1 



Mar 9 I no I 
Tota I s 



Nom | na I 
Agreement 



Marg I na l 
Agreement 



L I ght t s 
Extens | on 
of Kappa 







♦ 








♦ 


Used 


1 




0 


0 


0 


♦ 
















Not used 


2 


♦ 


12 




1 


♦ 
















Other 


3 


♦ 


0 


3 


8 








♦ 








♦ 



Marginal Totals 



+ 
♦ 
♦ 



12 



39 



♦ 

4 



0 
49 
11 

*} 
60 



♦ 
♦ 
♦ 
♦ 
♦ 
♦ 

♦ 



.735 
.727 



0 .000 
.796 
.818 



,242 
,679 



9 

ERIC 



category Marginal Agreementl ,538 
Cohen's Kappal ,396 



267 



208 



H 



Table 129 

Phase IV • ,IGB-*£va|uat I on Project 
Descr I ot I vfe Stijdy - Spring iv/8 

Re! labllNtV or » Material! 
Observer /0 Expert 80 
School 801 Grade 2 
Week 0 Day 4 
DMP 



<x 



Material! Printed material 



Category chosen 
by Expert 80 



Cat. 



Category chosen 
by Observer 40 



1 



Marfa I na 
Total s 



Nomina I 
Agreement 



Marginal 
Agreement 



Llght»s 
Extens I on 
of Kappa 



















Used 


1 ♦ 


0 


0 


0 




0 


♦ 




♦ 














Not used 


2 ♦ 


1 


62. 


2 




65 


♦ 










♦ 






other ' 


3 ♦ 


0 


2 


17 




19 






♦ 












♦ 



.954 

.895 



0.000 
.985 
1.000 



/ 

f 806 
.86* 



Marg I na I Totals ♦ 1 



64 



19 



84 



\ 



3'3'J 



category Marginal Agreementi .662 
Cohen's Kappal .834 



9 

ERIC 



Table 130 



Phate IV - IGE Evaluation Project 
Descriptive Study - spring 19T8 

Reliability on Materials 
Observer 10 Expert 90 
School 801 Grade 5 
Week 0 Day 5 
QMP 



Material! Printed material 



Category chosen 
by Expert 90 



Used 

Not used 
Other 

Mara I ra | 



Cat • 



C^teqory chosen 
by Observer 10 



Maro I na 
Tota I s 



Nom | na I 
Agreement 




.815 
1.000 



Mar g | na I 
Agreement 



0.000 
• 815 



600 



L I ght * s 
Extens | on 
of Kappa 



• 306 



1.000 



Category Marginal Ag.eementl «<*72 
Cohen's Kappal «<t85 



271 



Table 131 

Phase IV - K»t£ Evaluation Project 
Descriptive Study - SP,rlng 1978 

Reliability on Materials 
Observer 20 Expert 90 
School 801 Grade 5 
week 0 Day 5 
DMP 



Material) Printed material 



Category chosen 
by Expert 90 



Used 

Not used 
Other 



Mere I na I 



Cat • 
* 



1 ♦ 

2 ♦ 
♦ 

3 + 



Totals ♦ 



Cateqory chosen 
by, Observer 20 

1 2 3 



./ 



0 
1 
0 



0 
26 
1 

27 



0 
0 
2 



Mara I na I 
Tota I s 



♦ 
♦ 
♦ 
♦ 
♦ 
♦ 
♦ 

♦ 
♦ 
♦ 



0 
27 
3 

30 



> 
V 
♦ 

♦ 

♦ 

♦ 
♦ 



Nom | na I 
Agreement 



,963 
• 667 



Marg I na I 
Agreement 



LI ght » s 
Extens I on 
of, Kappa 



0.000 ----- 
1*000 .630 
.667 .643 



category Marginal Agreementl .556 
> Cohen's Kappal .636 



Table 132 



Phase IV - IGE Evaluation Project 
Descriptive study - spring 197a 

»e| | ab l M ty on Mater l a I s 
Observer 30 Expert 80 
School 801 Grade 5 
Week 0 Day 5 
DMP 



Material! Printed material 



Category chosen 
by Expert 80 



Cat, 



Cateoory chosen 
by Observer 30 



Marg I na I 
Tota I s 



Norn | na I 
Agreement 



Mara I na I 
Agreement 



LOt's 
Exte s I on 
of Ka 'pa 



Used 

Not used 
Other 



Marginal Totals 



♦ 
♦ 
♦ 
♦ 
♦ 
♦ 

4 



0 
6 
0 



0 
15 

0 

15 



0 
0 
7 



♦ 
♦ 

4 
♦ 
4 
♦ 

♦ 
♦ 
♦ 



4 

0 4 
4 

23 4 

4 

7 4 
4 

444444* 
4 

30 4 

4 



,65? 
1 • 000 



0.000 
• 652 
1.000 



• 30<» 
1 .000 



44444444444444444444 



9 

ERIC 



Category Marginal Agreement? #B51 
Cohen 1 s Kappal f 526 



Category chosen 
by Expert BO 



Table .133 

Phase IV - I CiE Evaluation Project 
Descriptive Study - Spring 1978 

Reliability on Mater I a I s 
Observer **C Expert 80 
School 801 Grade 5 
Week 0 Pay 5 
DMP 



Material! Printed materia 



Cat t 







♦ 






Used 


1 




0 


0 












Not used 


? 




3 














Other 


3 




0 


1 
















+ + ♦ 
















Mara I na 1 


Total s 




3 


21 






♦ 







Cateaory chosen 
by Observer 40 



1 



0 
o 
6 



Merg I na I 
Tota I s 

♦ 0 «■ 
+ + 
+ 23 ♦ 

♦ ♦ 

♦ 7 ♦ 
+ ♦ 
♦♦♦♦♦♦♦♦♦♦ 

♦ ♦ 

♦ 30 ♦ 

♦ ♦ 
♦ ♦♦♦ 



Norn I na I 
Agreement 



.870 
.857 



LlQht'S 
Marginal Extension 
Agreement i of Kappa 



0,000 

.913 .565 
.857 ,821 



category Mflrolnal Agreement! 
Cohen's Kapral .6^0 



590 



ERIC 



Table 134 



Phase IV - IflE Evaluation Project 
Descriptive Study - Spring 1978 H 

G; 

Reliability on Mater I a I s 
Observer 20 Expert 90 
School 801 Grade 2 
Week 0 Day 5 
WDRSD 



Materlaii Printed material 



Category chosen 
by Expert 90 



Cat , 

ft 



Cateoory chosen 
by Observer 20 



1 



Mar g I na 
Tota I s 



♦ ♦ + + + + + + + + + + + + ++ + + + + + + + + + + + 4, + 

♦ ♦ 4 

U* cd 1+000*0* 

♦ 44 

Not used P 4 50 o 3 * 53 + 

4 ♦ ♦ 

Other* 34 i o 0 * 1 4 

♦♦+♦♦♦*♦** >4 4 444444444444444 
+ 4 

Marginal Totals* 51 o 3 * 5 4 

♦ 4 



4 
4 
4 



Nom | na I 
Agreement 



0.000 
0.000 



Mar 9 | na l 
Agreement 



0.000 
0.000 

.3.13 



Light's 
Extens I on 
of Kappa 



0,000 
• ♦059 



444444444 4 44444444*4444444444 



category m * r n I n d | Agree r.ontl .111 
Cohen'* Kappaj -.001 



01 <) 



ERIC 

hfiiinniffnrmnaij 



Table 135 



Phase IV - IGE Evaluation Project 
Descriptive Study - SPr InQ 1978 

Re I I ab I I I ty on Mater I a I s 
Observer 30 Expert 80 
School 801 Grade 2 
Week 0 Day 5 
WDRSD 



MaterlalJ Printed material 



Category chosen 
by Expert 00 



Used 

Not used 
Other 



Cat, 



Cateoory chosen 
by Observer 30 



1 



1 ♦ 

♦ 

3 ♦ 

4 



16 



1 

0 



Mara 



I Total * ♦ 



4 

25 

44 

31 



4 

4444 



0 
3 
3 



Tota I * Agreement 



4 
♦ 
♦ 
♦ 
4 
4 
4 



20 



4 

4 
4 
4 
4 
4 
4 



44444444 

4 ♦ 

17 31 6 ♦ 5* ♦ 

4 4 

lf 4,444,444444' 44444444444444 



• 800 

• 862 

• 600 



Marg I na I 
Agreement 



• 650 

• 935 

• 8 3 3 



u 1 aht » s 
Extens l on 
of Kappa 



,708 
,676 
,550 



Category Marginal Agre#»mpnti ,873 



Cohen's Kappa 1 , 6 7 ?: 



Table 136 



Phase IV - IOE Evaluation Project 
Descriptive Study - spring 1978 

Re I I ab I I I ty on Mater I a I s 
Observer <»0 Expert 80 
School 801 Grade 2 
Week o Day 5 
WDRSO 



Materiel! Printed material 



Category chosen 
by Expert 80 



Cat . 



Cateqory chosen 
by Observer 40 



1 



Mara I na 
Totals 



Norn | na I 
Agreement 



Marg j na I 
Agreement 



L I ght • s 
Extens | on 

of Kappa 



Used 

Not used 
Other 



Marg I na 



1+ 20 0 0 ♦ 20 ♦ 

♦ ♦ ♦ 
2 ♦ 1 28 0 ♦ 29 ♦ 

♦ ♦ ♦ 
3* 0 0 5* 5* 

♦ ♦ ♦ 

♦ + * 

Totals ♦ 21 28 5 ♦ 54 ♦ 

♦ ♦ ♦ 



1 .000 
,966 
1.000 



.952 
,966 
1.000 



1,000 
,920 
1,000 



Category Marginal Agreement} 
roKen's Kappal ,967 



,073 



Table 137 

ph*M IV - IOE Evaluation Projact 
OftcrlPtlve Study - soring 1*T6 

Ral lability on taarnar Movat 
Obsarvar 10 ,. % EKpart 90 
School 601 Orada * 
weak o Day * 
W0R90 



cx 



Category choian 
by ExPtrt 90 



Cat » 
if 



Catagory chotan by Obtarvar 10 
i 2 3 * 5 6 



44444 4444< 











Engeoad - 1 t tan 


1 




3 


Cngagad * oral 


2 


♦ 

4 


0 


Engagad - eovart 


3 


4 
4 


3 


hot angapad - 1 ntar l « 




4 
4 


0 


Not angagad - waiting 


5 


4 
4 


0 


Kot angagad -o## task 


6 


4 
4 


0 


Isona o# tna abova 


7 


4 
4 


0 


Marginal Tntalf 


4 
4 


6 






4 





4 

0 
2 

0 
1 



2 
2 
6 

0 
6 

1 

0 



Marginal 
Total! 



Lights 

Nominal Marginal Extant I on 
Agrttmant Agraamant o# k«pP« 




1 


0 


I 


0 


4 
4 


11 * 


.2*3 


,543 


,162 


0 


0 


0 


0 


4 
4 


2 + 


0.000 


• 286 




1 


1 


2 


0 


4 
4 


15 « 


• 400 


e 602 


,12* 


4 


X 


1 


0 


4 
4 


6 < 


► ,667 


• 857 


t 6lT 


I 


3 


I 


0 


4 
4 


12 4 


► ,250 


• 600 


• IB* 


0 


1 


6 


0 


4 
4 
4 


6 


> tTftO 




• 666 


0 


0 


0 


0 


4 


0 









Catagory Marginal Agraa^anti «*33 
Cohan n Kappai #276 



Table 138 

phaia IV • IGE Evaluation Projact 
OatcrlPtlve Study - spring 1978 

Pal lability on laernar Movaa 
Obaarvcr 90 Expart 90 
School 601 ortdi 8 
WaaW o Day 4 

WORSO 



Category eHoien 
by Expert 90 



Cat* 
* 



CataQory choaen by Obtirver 80 

1 2 3 * 5 6' 



LlO^t 'I 

Marginal Nominal Marginal txttna Ion 
Total a Agreement Agreement o# KapPa 



Engaged - wr 1 ttan 


K 




7 


2 


2 


0 


0 


0 


0 


4 
4 
4 


11 


Engaged * oral 


2 


♦ 

4 


0 


f 


1 


0 


0 




0 


4 
4 


2 


Engagad - covart 


3 


4 
4 


2 


1 


10 


0 


0 


1 


1 


4 
4 


15 


Not angagad * interim 


4 


4 


1 


0 


0 


3 


1 


1 


0 


4 
4 


6 


Not engaged - waiting 


5 


♦ 
♦ 
4 


0 


2 


4 


1 


0 


3 


2 


4 
4 


18 


Not angagad »o#f taik 


6 


4 

4 


1 


0 


1 


1 


0 


5 


0 


4 
4 


6 


None of the abova 


T 


4 


0 


0 


0 


0 


0 


0 


0 


4 
4 


0 






♦444444444444 












4 




Marginal Tota 


I • 


4 
4 
4 


u 


6 


18 


5 


1 


10 


3 


4 
4 


54 





1 .000 


• 543 


• 500 


.333 


• 437 


• 667 


• 633 


• 500 


• 500 


• 633 


• 449 








0t000 


• 083 




• 686 


f 800 


• 540 




0*000 





catagory Margin** A«reementi •555 
Cohan'l Kappal #368 



J 



Table 139 

Phase IV « IOE Evaluation project 
OeicMPtlve Study • SPr|nQ 1978 

pal lability on Learner Move* 
Obiarvar 30 Expert aa 
school 801 Orada i 
WQ«k o Day 4 
WORSO 



1 



oo 

00 



Ca'.egor/ choten 
toy Expert 80 



cat* 
a 



Rngaoed - written 
Engaged « oral 
Engaged - covart 
Not an&eged • interim 
Not anbaged -waiting 5 
hot engeged »oM teifc 6 
hone ol tha abova T 



Marginal Total* 



Cataoopy chotan by Obiarver 30 



♦ 

♦ 

♦ 
♦ 



17 



Marginal Nominal 
Total ■ Agreement 



light'* 
Marginal Extant Ion 
Agreement o# KePPa 



8 


0 


1 


0 


0 


0 


0 


♦ 
♦ 


9 > ♦ 


,88* 


1,000 


,869 


0 


2 


0 


0 


0 


0 


0 


♦ 
♦ 


2 ♦ 


1,000 


,333 


.1.000 


1 


3 




1 


0 


2 


0 


♦ 
♦ 


21 ♦ 


• 667 


,810 


,538 


0% 


0 


0 


1 


1 


0 


0 




2 ♦ 


,S0C 


,500 


,464 


0 


0 


0 


0 


5 


1 


0 




6 ♦ 


,033 


,857 


,811 


0 


1 


0 


1 


0 


6 


X 


♦ 


9 ♦ 


• 667 


1.000 


,606 


0 


0 


2 


1 


1 


0 


7 


♦ 

♦ 


U ♦ 


,636 


o727 


,680 



9 % ♦ 60 



category Marginal Agreementi .747 
Cohanta Kappai .681 



Table -140 

PHqm IV - I3E ^valuation Project 
DatcrlPUva Study • Spring ivr* 

Ral lability on Learner Movao 
Obeerver 40 t Expart 00 
School eoi ■ orada 2 

Week ,0 Day 4 
WDRSD 



Catagory chotan Cat* 
by E*part BO * 




C*taoory 


choaan 
3 


by 

4 


Obtarvar 
5 6 


40 

T 


Marginal 
Total* 


Nominal 
Agraamant 


Marginal 
Agraamant 


























Enoagad • *r 1 ttan 


1 


♦ 
♦ 


i 

6 


0 


1 


0 


0 


0 


0 


♦ 9 ♦ 


• 669 


• 900 


Engaoad * oral 


2 




0 


1 


I 


0 


0 


0 


0 


♦ 2 ♦ 


• 500 


• 500 


fngagad • covart 


3 


♦ 
♦ 


\ 


3 


IT 


0 


0 


0 


0 


♦ 21 ♦ 


• 010 


• 913 


Kot angagad • Intarim 


4 


♦ 
♦ 


0 


0 


I 


J 


0 


0 


0 


♦ 2 « 


,,£00 


• 500 


Not angagad • waiting 


5 


* 


0 


0 


0 


0 


4 


2 


0 


♦ 6 « 


• 667 


,833 


Kot angagad ~o'* talk 


6 


♦ 
4 


0 


0 


1 


0 


1 


6 


1 


♦ 9 4 


,667 


• 869 


Nona of tha abova 


7 


♦ 


1 


0 


2 


0 


0 


0 


e 


♦ li < 


► f 727 


#616 



Marginal Total* ♦ 10 4 



S3 



60 

♦♦♦♦♦♦♦♦ 



LIOHt'S 
F.wtaniloa 

of k«p?>® 



• 867 

„$9l I; 

• *92 

• 636 

• *1« 

• *T9 



cat««ory Marginal Aaraananti *Y68 
COhama Kappal #680 



2K'6 



X) 



ERLC 



1. 



> 



Cngagad - wr I tten 
Engaoad « oral 
Engaged 9 eovfirt 
hot angaoad 



Table 141 

Phaia ;v • IDE Evaluation Project 
OaacrlPtlve Study - Sci-iM i 

fl«Mabl'l Itv on Learnar, Movai 
'Obiarvar~/0 Expart 80 
School 601 Orada 9 
Weak 0 Day * 

DMP 



O 



/ 



Cataoory chotan by Obtarvar 40 



Cattgorv cHotan Cat* 
by Entpart eo > 



Marol nal 
Total • 



Nominal 'Marginal 



Lights . 
Extant Ion 



Agraamant Aoraawant ol KaP*a 



\ ♦ 

♦ 

3 ♦ 
♦ 

|ntajj^!m 4 ♦ 
Not angagad -waiting 5 ♦ 
Not anoagad «oM task 6\* 
Nona of tho abova 7 ♦ 



Marginal Totals ♦ 
♦ 



3 


0 


5 


1 


1 


0 


0 


♦ 10 ♦ 


tSOO^ 


.500 


,256 
























1 


1 


1 


*v 


0 


0 


0 


* 3 ' ♦ 


• 333 


,3T5 


,263 


0 


5 


IT 


1 


6 


l 


0 


♦ 30 ♦ 


,567 


,e?3 


,363 


.0 


1 


0 


1 


0 


i 


0 


♦ 3 ♦ 


• 333 


1.000 


*30* 


* 

0 


0 


2 


0 


9 


l 


0 

i 


♦ 12 ♦ 


,T50 


,66T 


,662 


1 


1 


0 


0 


1 


2 


2 


♦ T ♦ 


-,266 


»esT 


1 ,231 














* * 




"T 




0 


c 


0 


0 


1 


1 


IT 


♦ 19 * 


*89S. 


1.000 


,16* 






*• 


















* 


ft 


25 


3 


18' 


a 


19 


♦ e* ♦ 




























y 


















f 




\ 




Category 


Marginal 


Aertamant t 


,T»T 


r 


) 





CoMnn Kappa I 0 »4»Q 

(.• O J 



9 

ERIC 



Table 142 



phase :v - tor r. . ■ - . \ . ict 
Descriptive Study - Spring i*/6 

Reliability on Learner Moves 
Obtarvar 20 Expert 60 
School 601 Grade 8 
week o 1 Day 4 

DMP 



Category chosen Cet» 
by Expert 60 a* 



Category chosen by Observer "20 

18 3^567 

♦ ♦♦♦♦♦♦♦♦4*,+ ♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦ ♦ 

♦ 



Marginal 
Totals 



Engaged • written 
Engaged - oral 
Engaged • covert 
Not engaged • Inter i 



\ ♦ 

♦ 

3 ♦ 

* *i 

Not «noag«d •waiting 5 ♦ 

Not onoaood »of# tank 6 ♦ 

♦ 

Nona of th« abova 7 ♦ 

i > 



9 
1 
3 
0 
3 
1 
0 



♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 

Marginal Total • ♦ IT 7 15 5 7 l* ' 19 
♦ 

♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 



0 


' o • 


l 


0 


0 


♦ . 10 • 


♦ .900 


• 588 


,8-75 


0 


0 


l 


0 




3 


♦ i333 


«ft29' 


,2*J 


14 


1 


i 


5 


0 


♦ 30 


♦ ,067 ^ 


,500 


,351 


0 


a 


0 


1 


0 


♦ 3 


» ,667 :V 


• 600 


,6ft6 


1 


i 


* 


3 


0 


♦ 12 


♦ .33.3 


• 583 


,273 


0 


0 


0 


5 


1 


♦ 7 


» -,71ft 


• 500 


,657 


0 


i 


0 


0 


18 


♦ 19 


» ,9*7 


1*000 


,932 



44 



Nom Ipa | 
Agreement 



Marginal 
'Agreement 



.Llgnr»s 
Extens Ion 
0$ KapPa 



category Marginal Agreement, ,600* 
Cohen t i Kappa t #sgs 



9 

FRIC 



290 



•nit 



it 



J' 



i; 



Table 143 



> 



phaia IV - IOC Eva motion Projec* 
Descriptive Study - Spring 19TB 

Reliability on Learner Movet 
Obterver 30 Expert 90 
school eon Grade St 
Week 0 Day 4 
OMP 



Category ehoten 
by Expert 90 



Category chosen b/ Obterver 30 




Engaged • written 


1 


♦ 

♦ • 


10 


0 


2 


0 


Enge&ed • oral 


2 


♦ 


0 


a 


1 


0 

1 


Cheesed - covert 


3 


♦ 


5 


3 


16 


0 


Not Ageoed « interim 




♦ 

♦ 
♦ 


0 


1 


1 


10 


Not engaged «■ waiting 


5 


0 


0 


3 


0 


Not engaged -of* teak 




♦ 


1 


1 / 


3 


- 2 


None o# the above 


7 




0 


i 


0 


4 



0 1 
0*0 



0 
0 

1 

0 
0 



0 

1 

1 

0 



1 

0 
0 
0 

t 

1 

3 



Marginal 
. Total* 



Hargmel Toteit ♦ 

• ♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 



* 16 e 26 16 1 • T 



TO 

♦♦f ♦♦♦♦♦ 



Nomine | Heroine I 
Agreement Agreement 



Lights 
Extension 
o# kappa 



1* ♦ 


,7i* 


,875 


,6*1 


3 ♦ 


,66T 


,378 


,629 


86 ♦ 


,6*0 


1962 


,460. 


12 ♦ 


.833 ' 


• ^780 


- ,790 


T ♦ 


, 1M1 


,1*3 


,132 


,9 ♦ 






,063 


6 ♦ 


,378 


,679 


,313 



i 



Category Merglnel Agreement! #632 
Cohen tt Keppai .Ml 11 



.291 



Table 144 



Phase IV - IOE Evaluation Project 
Descriptive Study, • •SP-.fltf ' 



Reliability on Lfc^rner Moves 
Observer 40 Expert 90 
School eoi t Orada ^ 
week o % Day 4 



Cotaoory chosen 
by Expert 90 



Engaged • wr I tten 
engaged • oral 
engaged • covert 



Cat» 



Category chosen, by Observer 40 
i 1 )a 3 h 5 6 T 

♦♦♦♦♦♦♦♦♦ 



Marg I na I Nom I ni | Mar g I na I 
Tote | s Agreement Agreement 



Light's 
extension 
of KapPe 



A ♦ 



3 ♦ 

/ ♦ 

Not engaged • Interim*. A ♦ 

♦ 

Not engaged • waiting S ♦ 

♦ 

Not engaged •off task 6 ♦ 

Nona of the above 7 ♦ 

♦ 



Marginal Totals ♦ 



♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦ ♦♦♦♦♦♦ 



12 


0 


2 


0 


0 


0 


0 


♦ 1* ♦ 


.857 % , 


- ,824 


• 817 


* 0 


1 


1 


0 


0 


0 


. 0 


♦ 3 ♦ 


' .«67 , 


,500 


,639 


4 


1 


1' 


1 


. o 


0 


0 


♦ 25 ♦ 


,760 


• 862 


,618 


1 


1 


1 


9 


0 


0 


0 


♦ It * 


' .750 


1*000 


,705 


0 


0 


3 


1 


2 


i 


0 


♦ 7 ♦ 


.206 


,886 


,267 


0 


1 


3 


i 


0 




0 


♦ 9 t* 




,556 


,606 


0 


1 


0 


0 \ 


0 


0 


7 


♦ 8 ♦ 


• 875 


,678 


,663 




















♦ 




IT 


6 


29 


12 


2 


s 


7 


♦ 76 ' ♦ . 







♦♦♦♦♦♦♦♦ 



ERIC 



category Marginal Agreement* #700 
Oohanrs Keppat t«30 | 



n r j 



232 



ft 



4 

s 

Table ,1^5 f 

phasa IV - 10E Evaluation Projact 
OatcrlPtlve Study - Spring 1978 

J, 

Reliability on Laarnar Movai 
Obaarvar 10 Expart 90 
School SOI Orada48 
y Week 0 Day S 

DMP 



\ 



Catagory c^ottn. 
bycxpart-90 ^ 



Cat. 
# 



I 
2 
3 



Engaged - wr 1 1 tan 

<•> 

Engaged - oral 
Engegad - eovfcrt 
Not engaged •» |ntar|m> * 
Not angagad -waiting 5< 
Not angagad -oM task 6 
Nona of th* abova 7 

Marginal Tptali 



Category chotan by ObMrver/10 



I 2 3 * 
♦ ♦♦♦♦ 



ar/io 

6^, 



•Marginal 
T0ta|« 



LlOht'l 

Nominal MarOlnal Extanilon 
Aoraamant acraamant ©♦ Kappa 



7 j 9 1 * 5 5 ♦ 30 

* 



3 / 


0 


5 


1 , 


0 


4 

i 


1 




11 ♦ 


• 273 


,636 


,051 


0 


0 


0 


0 


0 


0 


0 


♦ 
♦ 


0 ♦ 




0,000 




3 


I 


* 


0 


0 


0 


1 


♦ 
♦ 
♦ 


9 ♦ 


,**«♦ 


1,000 
0,000 


,20« 


0 


0 


0 


0 


0 


0 


0 




0 ♦ 






0 


0 


0 


0 


a 


. o 


0 




2 ♦ 


1,000 


1,000 


1,-000 


1 


0 


0 


0 


0 


.. * 


0 






• S00 


1,000 


• ,T60 


0 


0 


0" 


0 


0 


0 


3 


♦ 




1,000 


,•00 


1,000 



Catagory Marginal^ Agreamentl; «60S 
Cohamt Kappal #399 

293 



ERIC 



, Table 146 

phase XV - IGC Evatuat Ion Project 
OaicriPtlve study • soring 1976 

X) 

Reliability on ^earner Moves 
Observer '20 1 Expert 90 
School 001 'tirade 9 
Week o Day 5 

DMP 



.Category chosen 
by EKPart 90 



Enaaoed - written 
engaged - oral 
Engaged - cover t f * 
Not engaged - inte 



Cat. 
* 



None of the above 



Category chosen by observer 20 







a 


3 


4 


5 


6 


T 




Totals 
























♦ 






0 


2 


0 


0 


0 


0 


♦ 


11 ' 






f • 




















♦ 


a 


0 


0» 


0 


0 


0 


0 


♦ 


0 


♦ 


♦ 










1 






♦ 




♦ 


♦ 


6 


1 


1 


0* 


0 


0 


♦ 


9 


♦ 






















♦ 




• 


o" 


0 


0. 


' 0 


0 


0 


♦ 


0 


t 


















» ♦ 






♦ 


0 


0 


0 


0 


2 


0 " 


0 


♦ 


2 




♦ 
















♦ 




♦ 




2 


0 


1 


0* 


0 


2 


0 


♦ 


8 




♦ 




* 












♦ 




♦ 




0 


0 


0 


0 


1 


0 


2 




3 




<> 
















♦ 






♦ 


















♦ 


♦ 


IT 


1 


' * 


0 


*> 


2 


2 


♦ 


30 


♦ 
















♦ 




♦ 

























Marginal NonHnai Marginal 
agraamant agraamant 



Lignt'l 
Extant Ion 
of KaPPa 



.818 



,64T 
OiOOO 



,580 







• ,026 








1,000 


,900 


T.000 


♦ *oo 


,400 


,3BT 




,»*7 


,643 



Catagory Marginal aoraamantl »a<>3 
CoHanta Kappal ,3S7 



'M3 



0 

ERIC 



294 



Table 147 

Phata IV - IOE Evaluation Projact 

utticr i .stive' Study - ./pmi.vi 

Rell-ablHty on learner- Movei 
Observer 30 Expert SO 
School B01 , Orada S 
week o Day 5 

OMP 



Cataoorv chosen 
by Expart 80 



Cat* 
a 



Cngagod - writ tan 
JEnoagod - oral 
Cnoaged • covert 



1 ♦ 
- ♦ 

2 ♦ 
♦ 

3 ♦ 
♦ 

Not engaged • Interim * ♦ 
Not engaged «• waiting 5 ♦ 
hot engaged «of# tasV 6 ♦ 

T ♦ 



Nona o# the pbove 



Category chosen by Observer 30 

I 8* 3 * * 6 



5 

0 
1 

0 
, 0 
0 
0 



0 
0 
0 
0 
0 
0 
0 



2 

« 

0 
0 

1 

0 
0 



0 
0 
0 

1 

0 
0 
0 



0 
0 
0 

1 
s 

0 
0 



Marginal Totals ♦ 6 0 7 i > 6 
♦ 



c 

\ 



0 

0 * 



0 

1 
1 

0 



0 
0 
0 
0 
0 
0 
7 



Marginal 
Total i 



♦ 



30 



Nominal Marginal 
Agreement Agreement 



Llflht's 
Extorts I on 
of Kappa 



♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 



T 


♦ .71* 


,857 >■ 


,6*3 


0 
6 


♦ .667 


,857 


,865 


2 


♦ .500' 


,500 < 


,463 


T 


♦ nTi* 


,657 


,6*3 


1 


♦ l.Ooo 


,333 


1,000 


7 


♦ i«ooo 


1,000 


1.000 



■ V 



Category Marginal Agreementi #73* 



Cohen's Kappa I .708 

295 



ERIC 



K 



V 



Table 148 

Phaia iv - IOC Evaluation Projtct 
DaicrlPtlva Study • spring 19T8 

Reliability on L«ernar Movai 
Obtarvar *o Expert 80 
School 801 Orada 9 
Waak o Day 5 

DMP 



M 



Catagorv chotan 
by Expert 80 



Catt 
a 



Cataoory chotan by Obtarvar 40 



♦♦♦♦♦♦♦♦♦♦♦♦ 
♦ 



1 ♦ 
\ ♦ 

2 ♦ 
♦ 

3 ♦ 
♦ 



Engagad - writ tan 

Enflagad ~ oral 

Engagad ■» covart 

Not engaged « Interim * ♦ 
' * ' ♦ 

Not angegad -waiting 5 ♦ 

♦ 

Not engagad «o«K t ** k 6 ♦ 

♦ 

Nona of the above 7 ♦ 

♦ 



Marginal 
Total ■ 



Marginal Total* ♦ 
♦ 




Nominal Marginal 
Agreement Agreement 



• 85? 

• 633 
.500 

• 057 
1.000 

• 817 



1*000 
0*000 
1*000 
,500 
1*000 

• 500 

• 867 



ligfc|" 
Extension 
of KaPPa 



♦ 81* 



.792 
,*83 
.81* 
1.000 
• 821 



Cataoory Marginal Agraamant t ,#69* 
Cohftntt Kappat #791 



Jo 



ERLC 



Table W 



Phase XV 

Dt^cr iPt I ve 



- IQE Evaluation Project 
Study SPrinfl 197ft 



Rellabl I Itv^on uearner Moves 
Observer 10 Expert 90 
school 801 Oreda 8' 
week 0 Day 5 
WORSO 



"category cho'sen 
by expert 90 



It 



engaged • written 
fngaged • oral 
engaged • covert ' 



cett 
* 



Category chosen by Observer 10 
\ 2 3 * * * 7 



Marginal Nominal .Marginal 
Totals Agreement /Agreement 



Wight's 
extension 
of KapPa 



1 ♦ 
♦ 

2 ♦ 
♦ 

3 ♦ 
♦ 



Not angaged - interim 4 ♦ 

hot ergaged waiting ♦ 

♦ 

Not engaged »oM task * ♦ 
None o* the above T ♦ 

♦ 

Marginal Totals ♦0 1, 



,0 
0 
0 
0 
0 
0 
0 



0 
0 
0 
0 

4 1 

0 
0 



0 


0 


0 


0 


0 ♦ 


0 ♦ 
















♦ 


0 ♦ 




0.000. 




0 


0 


0 


0 


Or* 










11 


0 


1 


0 


♦ . 
1 ♦ 


13 ♦ 


,0*6 ^ 


* ,765 


.775 


1 


6 


1 


0 


O ♦ 




.750 


,075 


.713 


8 


1 


21 


1 


♦ 

1 ♦ 


30 ♦ 


.700 


,600 


,460 


0 


0 


.1 


1 


♦ 

0 ♦ 


2 ♦ 


• 500 


1,000 


i. * \ 


0 


• 0 


0 


0 


1 ♦ 


1 ♦ 


1.000 


,333 


1,000 










♦ 


















▲ ♦ 








17 


7 


2* 


2 


3 ♦ 
♦ 


54 ♦ 




1 





C-t-eorv M-pgin«| Aor-tm.ntl ,62* 
Ceh«n*t k«pp-» .001 
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lable 150 
t 

Phe*e IV - IGE Evaluation 'Project 
DeeeriPtlve Study - spring 1976 » 

Reliability on-teerner Movet 
Observer 20 Expert to 
School eoi Grade 2 
"waal* o f Day 5* r 

WORSD 



Category chot'en Cet# 
by Expert 90 * 



Engaged - written 
Encaged • oral 
Engaged - covert 



Category- chotan by Obiarvor So 

1 2 3 k 5 6 7 

♦ *♦♦♦♦♦♦♦♦*>♦♦♦♦♦ ♦♦♦♦♦♦♦♦♦♦♦ 

♦ 



1 ♦ 

2 ♦ 
♦ 

3 ♦ 
♦ 

Not engaged •» Interim * ♦ 



Not engaged •waiting 5 a 

♦ 

Not engaged «bf# task 6 ♦ 



None of • the above 



0 
0 
0 

1 
i 

0' 
'J 



0 
0 

1 

0 
0 
0 
,0 



0 
0 
8 

t 



0 
0 



0 
0 

0 

s 
I 

% 0 

1 



0 
0 
3 
2 

tl 
1 
0 



0 
0 
0 
0 

I 
1 

0 



0 
0 

I 

0 
2 
0 
0 



♦ 

7 ♦ 
♦ 

' ♦♦♦♦♦♦♦♦♦♦♦ 4 ♦ ♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦>♦♦♦♦♦♦♦♦♦♦♦' 
♦ 

Marginal Totals ♦ 2 1 12 7 27 2 3 

+ * + + + * + + + + + + + + + + + + + + + + + + + + + + + + + + + + + 



Marginal 
Toteii 

0 

* 0 

e 

30 
2 

♦ ♦♦♦♦%>«>♦ 
♦♦♦♦♦♦♦♦ 



Nominal 
Agreement 



#615 
#625 
#700 
#500 
0.6O0 



0#06< 



Marginal 
aoreamant 

0r000 
OsVoO 

• 929 
,8TB 

,900 

wooo 

♦ 333 



l.|ght'i 
Cxtana Ion 
o» KaPPa 



,506 
,400 

,aei 

\ 



Category Marginal Agreement I #676 
COhenti Kappa t #«*96 
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Table 151 



\ 



Phaie IV - IOE Evaluation Projept 
-OaicriPtlva Study - SPrino 19TB 

Reliability on Learner Movea 
Obierver 30 Expert 80 
"school etfl Grade 8 
Meek, 0 Day 5 
WDRSO 



\ 



\ 



Category c^oien 
by c Expert 00 



Category ehoien 



♦ 

1 ♦ 
♦ 

2 ♦ 

3 ♦ 

♦ \ 

Not enoaoed'- Interim k ♦ 

/ ^ f ♦ 

Not engaged - welding * ♦ 
Not engaged -of f )taiM 6 ♦ 



Engaged • written 
Engaged - oral 
Engaged m covert 




^ Observer 30 

i 5. 6, T 



V No4* of*the above 



♦ 

7 ♦ 

Heroinal Totaia ♦ 15 2 



?" 


1 


* 1 




1 


0 


1 


1 


2 


0 


0 


0 


2 1 


9 




* 0 


2 


3 


1 


0 


0 




. 0 


0 




0 


0 


' 1 . 


2 


0 




•> 












•fi 


2 


2 


0 


6 


0 


0 


> 


i 


0 


0 



Mara fine I 
♦♦♦♦♦ 

0 .♦ 

'•" ♦ 

o^ ♦ 
■ ♦ 



i 
i 

0 
3 



♦ 
♦ 
♦ 
♦ 
♦ 

♦ 
+ 
♦ 



2 
5*. 
12 



\ 



■Nominal 
' agreement 



Harof na-1 
agreement 



LIOHt'f 
txtentlon 
o# KaPPa 




dateoorv Marginal aaceement I i732 
Cohen'i Keppai .38»< 



20 J 









,750 


' ,sdo 


,66* 


,250 

> 

,429 


,500 


,221 


,85T 


,265 


'0*000 


,500 


• •080 


,*00 * 


1,000 


,339 


♦5oa 


,780 


,*00 


,*oo 


,71* 


*8*0 



C 

c 



/ 



9 

ERIC 



Table 152 

Phase IV * IGE Evaluation Project 
Descriptive study Spring i$?a 

'/ 

Reliability on Learner Movet « 
Qbnerypr 40 Expert 80 
School eoi Grade 3 

Week 0 Day s \ 
\ WDRSO 



Category choten Cat, 
by Expert ,80 / n 



Category choten by Observer *o 
3 * 5 6 7 



1 



< 

2' 



♦♦♦♦4 ♦♦♦♦♦♦♦ 



^noegfd - written 


1 ♦ 


10 


0 


i 2 


0 


0 


0 


0 i 


► 12 ♦ 


• 833 


a 750 


,763 


Engaged • oral * 


♦ 

a ♦ 


0 


3 


1 


0 


0 


0 


o -/< 

* 


► . * ♦ 


.750 


1.000 


,730 


Engaged • covert .* 


3 ♦ 


1 


1 


7 


i: 


0 


0 


0 < 


► »* ♦ 


.500 


1*900 


„32S 


Not angaged •Interim 


0 


0 


0 




0 \ 


0 


0 < 


a 


1.000 


. .286 


1,000 


r ^ot engaged - waiting 


♦ i 


- 0 


0 


0 


1 




0 


6 < 


- s ♦ 


• 600 


, *800 


,78* 


Not engaged ~o9f task 


6 ♦ 


3 


0 


* 


J 


0 


•> 


" 0 - 


► 18 ♦ 


• 333 


• 333 


,2ft0 


None of the above 


♦ 

• 7 ♦ 


0 ' 


0 


0 


0 


0 


p 


5 < 


► ' *5 ♦ 


1*000 


1.000 


1,000 



Marginal Totelt ♦ 16 4 U 7 4 * 8 
♦ 



Marginal 
7o t a 1 1 



♦ 

♦♦♦♦♦♦♦♦♦ 



< LlOHt*! 

Nominal Margins) Extanalon 
Agp«*mant Agraamant ©♦ KePPa 



cataoorv Marginal Agraamanti 
+ Oohan^i Kappai ,873 



.738 



© 



9 
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Table 153 

phase IV • IG£ evaluation Project 
Descriptive study • spring 1978 

Reliability on Learner Novel 
Observer lo • expert SO 
School 601 Orade i 
Week 0 , Day 8 

OMP ' 



o 



Ceteoory choien C*t# 
by expert 60 * 



Category chosen by Obierver lo 

^23 * S 6 7 

9 

♦ ♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 



Marginal 
Totals* 



Nominal 
Agreement 



Marginal 
Agreement 



Lights 
Extension 
o* Kappa 



Engaged • written 


1 




4 


0 


0 


1 


0 

y 


0 


0 


4 
4 

4 


9 ♦ 


• 600 


,625 


#.777 


Engaged • oral 


2 


♦ 


0 


3 


0 


0 


0 


0 


-0 


4 
4 


3 ♦ 


1.000 


,800 


. 1.000 


Engaged « covert 


3 


♦ 


3 


2 


2* 


10 




1 


2 


4 
4 


*6 ♦ 


.522 


• 56T 


,269 


Mot engaged • Interim 




4 


0 


i 


0 


2 


1 


0 


0 


4 
4 


* ♦ 


• 500 


• 266 


,391 


Not tfngaged • waiting 


S 


♦ 


0 


u 


2 


0 


6 


0 


3 


4 
4 


11 ♦ 


♦ 5*5 


l.'ooo 




Mot engaged «oM task 


6 




0 


0 


1 


0 


0 


3 


0 


4 
4 


* ♦ 


• 780 


1*000 


,736 


None of the above 


7 


4 


1 


0 


0 


1 


0 


0 


3 


4 

* 


8 ♦ 


• 800 


• 895 


,85* 








4444444444444444444444444444444444444444444 








Marginal Totals 


4 
4 
4 


6 


6 


27 . 


\* 


11 




8 




76 ♦ 


0 -s 







4444444444444444444444444444444444444444 4 >444444 



category Marginal Aoreementi •••Q 
Cohento Keppai «*3* 
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; Table 154 

Phaia IV • IOE Evaluation Projaet 
Deicr lPtlve,Study - SPrin* 1978 

Ret lability on laernor Mdvas 
Observer 20 Expert SO 
school' 801. Grade 2 
week 0 Day 5 
OMP 



Category chosen 
by Expert 60 



.Cat, 



Category chosen by Observer 20, 
I l ' 3 4 5 6 T 
♦ ♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 



Marginal 
Total c 



Nominal MerOI,nal 



tlOht't 
Extension 



Agreement Agreement of KaPPa 



/ 



Engaged ■ written 


I 

r 




2 


0 


2 


1 


0 




0 


♦ 
♦ 
♦ 


5 ♦ 


• 400 


.800 


,368 


Engaged • oral 


► 

2 


♦ 


0 


3 


0 


0 


0 


0 1 


0 


♦ 
♦ 


3 ♦ 


l.ooo 


,429 


1,000 


Engaged - covert 


3 




2 


2 


38 


0 


2 


1 


1 


♦ 
♦ 


♦6 ♦ 


.826 


,958 . 


,5*6 


Not engaged - Interim 


* 




0 


i 


2 


1 


0 


0 


0 


♦ 


* ♦ 


,250 


• 500 


,230 


Not engaged « welting 


5 


♦ 


0 


i 


3 


'0 


6 


0 


1 




U ♦ 


.8*6 


• 909 


.679 


Not engaged -off task 


6 


♦ 
♦ 


0 


0 


1 


0 


1 


2 


0 




k ♦ 


.500 


• 750 


,#00 


None of the above 


T 


♦ 


0 


0 


2 


0 




0 


2 


♦ 
♦ 


.5 ♦ 


»*00 


• 800 


,369 






♦ 












>*♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 








Marginal Totals 


♦ 
♦ 


k 


7 


48 


2 


10 


3 






7« ♦ 









Category Marginal agreement! .735 
cohemt Kappa t f *98 



N3 

o 
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Table ,155 

Phftit IV - lOt Evaluation Projact 
DateriPtlva Study - Sprang 19TB 

Reliability on Ltarnar Movat 
Obiarvar 30 Expert 90 
School 601 Orada a • 
waok 0 Oay 5 

DMP 



ho 
O 



4 ■* 

Cataoopy eheian by ObMrvar 30 



Cat* 
n 

♦ 
♦ 

* l ♦ 

2 ♦ 

3 ♦ 

Not angagad « intarim * ♦ 

♦ 

Not angagad • waiting- 5 ♦ 
Not angagad «v<>M talk 6 ♦ 
Nona of tha abova T ♦ 



Category e*otan 
by Expert 90 



Engaged •» writ tan 
Engaged • opal 
Engaged • covert 



Marginal 
Total! 



I i 3 . s * T . 



, Nominal Marginal 
Agreement ■ Aoraamant 



Light'* 
txtonilon 
o# KaPPa 



T 
0 
5 
3 
0 



1 
1 
* 

0 

1 



0 
2 
10 

1 
1 

2 

0 



0 
0 
0 
T 
0 
1 
0 



1 

2 
0 
3 
0 
1 



0 

1 
1 
1 
1 

13 
1 



Marginal Totals ♦ 
♦ 



IT • T 16 • T 16 11 



♦♦♦♦♦♦♦♦ 
64 

♦♦♦♦♦♦♦♦ 



0 


♦ k a ♦ 


. ,875 


,471 


,6*3 


0 


♦ St 


♦ 200 


♦ 71* 


,187s 


0 - 


♦ 82 ♦ 


«*6S ' 


.787 


,386 


0 


♦ u ♦ 


,563 


,667 


,53« 


2 


♦ 7 ♦ 


,*89 


1,000 


,377 


r 
1 


I *19 ♦ 


me* 


,9*7 


,698 


6* 


♦ 11 ♦ 


<7«7 - 


1,000 


,666 



deteeory Marginal Agreewenti 
Oohanta KaPpal »S0* 



,769 



9 

ERIC 
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Table 15 ft 

Phase XV - |GE Evaluation Project 
Deter iPtlve Study - Sor I'no 1976 

Reliability on Uamar Move! 
Obiervari^ o Expert 90 
School noi Grade 2 
weeK o Oay s 

OMP 




Catagory ehoien 
by expert 90 



Encaged • wr I t tan 
Cngagad * oral 
Engaged « covert 



Cat* 
a 



Catagory chose* by Observer 10 
1 2 3 * 5 * T 



Marginal 
Totafi 



„ ♦♦♦♦♦♦♦♦♦♦t ♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦ ♦♦♦♦♦♦♦♦ 

• V 



Nominal Herglnel 
Agreement Agreement 



Lights 
Extension 
of KapPa 



1 ♦ 
♦ 

8 ♦ 

3 ♦ 



Not engaged - interim * ♦ 

♦ 

Not engaged.- weltlpg 5 ♦ 

♦ 

Not engaged «off talk 6 ♦ 

None o* the above 7 ♦ 

♦ . 

♦ 

Hkroinal Total • * IS 



a 

o 

5 
1 
0 
1 

0 



3 
0 
1 
0 

1 



0 
2 
12 
2 
2 
0 
1 

19 



0 
0 
0 
7 
0 
0 
0 



► ♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 



0 


0 


0 


♦ 

♦ 


a ♦ 


i«ooo 


,533 


1 (000 


0 


1 


' 0 


♦ 


8 * 


• 400 


,n* 


,3*5 


0 


2 




♦ 


22 « 




,66* 


,*13 


0 


2 


0 


♦ 


12 < 


► .583 


•,563 


,5*5 


2 


1 


1 


♦ 
♦ 


7 H 


► t 266< 


#286 


.268 


0 


IB,. 


0 


♦ 
♦ 


19 < 


► «9*7 


.760 


,925 


0 


1 


a 


♦ 
♦ 
♦ 


iy < 


► ,727 




,6*5 


2 


25 


9 


♦ 
♦ 


64 
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Category Marginal Agreements t6Sl 
Cohen »t Kappa l #aio 
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o 



V 
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Table 157 

Phase IV- IOE Evaluation Project 
DtaerlPt.lve study - spring l»Ti 

Pal lability on interactions 
Observer 10 Expert 90 
School 801 Orada 2 
Week 0 Day 4 
WORSO 



5 



Oatfl 0*-25*T9 
Program! RELP*S 
Data ri lai KElDl 



Category choien 
by E*P6rt 90 



Cat, 
a 



Tercet to" TancHtr 
Target to Other^ Adul t 
Taroat to Student 
Target to Tchr I orp 
Teacher to Taroat 



1 ♦ 
♦ 

3 ♦ 

4 ♦ 
♦ 

5 ♦ 
♦ 

Ctnar Adult to Taroat 6 ♦ 
9tudant v to Target 
Teacher tc Group 
Other Adult to oroup 
'student to orouo 
Nona of the -above 



e ♦ 

9 ♦ 
♦ 

10 ♦ 
♦ 

U ♦ 



Marginal Totaia ♦ 



Category chotan by obterver- 10 



Marginal . Nominal 



1 


2 


3- 


4 


5 


6 


T 




a 

w 


10 


11 


"' Totaia 


I 
- 






























A 

V 


A 
V 


0 ' 


0 


0 

* h 


0 


0 


1 


0 


0 


>1 


♦ 
♦ 


2 




0 


0 


0 . 


0 


V 

0 


0 

f 


'0 


A 

V 


A 

V 


o 


0 

• 


♦ 
+ 


. o 




* 

0 


0 


1 


0 


0 


0 


0 


A 
V 


o 


o 


1 


♦ 
♦ 


2 


♦ 
♦ 


0 


o* 


0 


0 


0 


0 


0 


0 


0 


0 


0 • 


♦ 


0 


♦ 
♦ 


0 


0 


0 


0 


0 


0 . 


0 


0 


0 


0 


0 


♦ 


0 


♦ 


-o 


0 


0 < 


0 


0 


0 


0 


0 


0 


0 


0 


♦ 


0 


♦ ■ 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 


♦ 
♦ 


0 


♦ 
♦ 


I 


0, 


0 


0 ^ 


0 


0 


0 


4 


0* 


0 


0 


♦ 


s 


♦ 
♦ 


0 


0 


0 


0 




0 


0 


0 


0 


0 


o. 


♦ 
♦ 


0 


♦ 


0 


0 


, 0 


0 


0 


0 


0 


0 


0 


' 0 


0 


♦ 
♦ 


0 


♦ 


2 


1 ■ 


3 


3 


0 ' 


0 


0 


6 


0 


0 


- 30 


♦ 




♦ 
♦ 






























y 




4 


3 


0 


0 


0 


11 


0 


0 


32 


+ 


64 


♦ 



M»rglnm Ext*n*|en 
WPMMnt Aflr«em«nt o# k«ppq 



0^00 ' 



i5Q0 



,600 



• 66? 
0,000 

• 500 
0 9 000 



• 455 



# 667 



• ru 



-•0S9 
• 460 



,7*9 



.102 



') r r 



category Mara.lnal Agrt«m«nti «3fl» 



( 



Table 15 a 

phase IV - IOC Eveluatlon Project 
Descriptive Study V SPMng 1978 

Reliability on interaction! 
Obsarver 20 exp^t 90 
School 801 OrAde 2 1 
\ Waak 0 Day * 

x VlORSO 



Datel 06-25-79 
Program! reipqs 
Data file! RELPl 



Catagory chosen Cat, 
by Ewpart 90 * 



* Category ehoten by Observer 20 
4 H 5 6 7 8 9 10 II 



i* a 



Marginal 
Totals 

♦♦♦♦♦♦♦♦ 



Targat to. Taacneft 



1 ♦ 
♦ 

Targat to Other Adult 2 ♦ 

♦ 

Targat to Student * 



Targat to Tchr k Orp 
Teacher to Targat 



3 ♦ 
♦ 

4 ♦ 

5 ♦ 
♦ 



OtHar Adult to targat 6 ♦ 

♦ 

Studgnt to Targat 



7 ♦ 
♦ 

a ♦ 

9 ♦ 
♦ 

10 ♦ 
♦ 

11 ♦ 
♦ 

♦♦♦♦♦♦♦♦♦♦♦ 
♦ 

Marginal Total! ♦ * I 
♦ 



Taacher to oroup^ 
Othar Adult to Group 
Studant to oroup 
None 0* the above 



0 


1 


0 


0 


0 


0 


0 


I 


0 


0 


0 < < 


2 


























0 , 


0 


0 


0 


0 


0 


0 


0 




0 


0 '< 


> 0 


0 


0 


1 


0 


0 


0 


0 


0 

\ 

\ 


* 

0 

< 


0 


1 i 


► . ' 8 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 - 


► 0 


0 


o m 


0 


0 * 


JO 


0 


0 


0 


, 0 


0 


0 


» ' 0 


0 


0 


0 


0 


0 


0 


0 


0 


\0 


0 


0 ' 


» 0 


0 


0 


0 


0 


0 


0 


0 


0 . 




•0 


0 


» 0 


1 


0 


0 


0 


0 


0 


0 


3 


0 


.0 


1 


* ' 5 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 


♦ 0 


0 


0 \ 


0 


0 


b 


0 


0 


0 


1 0 


0 


0 


* 0 


3 


0 


1 ' 


0 


0 


0 


• 0 


q 


0 


0 


♦1 


♦ 49 



k © 0 - *3 • 



Nominal MarBlnai EKtamion 
Aeraemant Aoraanant. o# Kappa 



♦♦♦♦♦♦♦♦ 
5* 



0*000 



• 500 



■ 600 



.911 



,500 
0*000 
1.000 



,600 



,956 



► ,080 



,*6J 



,568 



,566 



301 



Category Maroinel Agreement! #681 
Cohen't Kappat 



308 



O 



O 
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* Table 159 

Phase IV -*IOE Evaluation Project 
DatcrlPtlve Study • Sprmo 1978 

Pel lab I I I fly on 
N Observer jIO 
School 601 
week 0 



Detal P6-25-79 
Prooraml RELP9S 
Data Fl |tl RELD1 



Interact Ions 
EKpart so 
Grade 2 
Day * 



Category chotan 
by E*P«rt 80 



Cat, 



HDRSD 

Category choaen by Observer 30 
3 4 5 6 7 8 < 



10 U 



Target to Teacher 
Target to Other^A* 
Target to Student 



♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 

C 0 



Marginal 
Total* 



Nominal 
Agreemant 



Marginal Extent ion 
Agreement .0* Kappa 



Target to Other Adult 2 ♦ 
v - ♦ 

3 ♦ 
♦ 

Target to Tchr I Orp * * ♦ 

♦ 

Teacher to Target S ♦ 

♦ 

Other Adult to Target 6 ♦ 

♦ 

Student to Target % 7 ♦ 

Teacher to Group S ♦ 

♦ 

pther Adult to OrouP 9 ♦ 

10 ♦ 

hone of the above sil ♦ 

♦ 



Student to Group 



0 
0 
0 
0 
0 
0 
0 
0 
0 
0 



0 
0 
0 
0 

1 

0 
0 

1 

0 
0 




399 ■ 



f 



Marginal Totalt ♦ 



category Margin*! Agreemanti #30* 
wohantf Kappal tT3« 



•0 

♦♦♦♦♦♦♦♦ 



0 


0 


0 


0 


0 


0 


0 • 


0 


0 


♦ 0 ♦ 






0 


0 


0 


0 


* 

0 


0 


0 . 


0 


0 


♦ 0 ♦ 




OrOOO 


3 


0 


0 


0 


0 


0 


0 


0 


0 


♦ 3 ♦ 


1.000 


,780 


0 


0 


0 


0 


' 0 


0 


0 


0 


0 


♦ o « 






0 


0 


0 


0 


1 


0 


0 


0 




♦ 1 * 


0*000 


0t000 


0' 


0 


0 


0 


k 


0 ' 


0 


0 


0 


♦ 1 < 


0*000 


0,0ff0 


0 


0 


0. . 


f 

0 


0 


0 


0 


0 


0' 


♦ 0 < 




0,000 


0 


0 


0 


0 


0 


a 


0 


0 


3 


♦ 11 < 


> ,78T 


,727 


0 


0 


0 


0 


0 


0 


0 


0 


0 


♦ 1 < 


► 0*000 


0,000 


0 


0 


0 


. 0 


' On 


0 


0 


0 


0 


t 0% ! 






1 


0 


0 


0 


' 0 


0 


0 


0 


*2 


♦ 43 ' 


y - ,»TT 


t 986V_, 



V 



,1,000 

0*000 
Of ono 

• ,668 
0,000 

,907 



*v X V 



\ 



Table 160 



Phase IV IOE Equation Project 
Descriptive 5tu4| spring 19TB 



Reliability orV interactions 
Observer*40 Expert So 
School 801 
Week 0 

WORSD 



drade 2 
Oey V 



Datet 06*25-79 
Progrefni RELP9S 
Oete F| let RELD1 



Category chosen Cat. 
by Expert BO * 



Category chosen by observer *Q 
I 2 3 4 ]5 6 T B t • 10 11 



Marginal 
Total* 



Target to Teachar 



♦ 

Terqet to Other Adult 2 ♦ 

♦ 

Target to Student 



3 ♦ 
a ♦ 
5 ♦ 

Other Adult to Target 6 * 



0 
0 

r 0 



Terfaat to TchV t or^ 

4 

Teacher to. Target 




Student to* Target 
Teacher to Group 
Other Adult to aroup 
Student to group 
Nonfc o# the above 



Nominal 
Agreement 



311 



Catagory Marginal Agraamantl 
Conan't Kappat »6to 



• BRT 



• 393 



1*000 
UOOO 

• 909 
0f000 

i tlftO 



Light** 
Marginal Extension 
Agreement ■ of Kappa 



(600 

1*000 
^»900 
0*000 
*917 

0*0,00 
,M6 



312 



,873 

1*900 
1*000 

*S66 
0*000 

• 619 



\ r 



'Category chosen Cet# 
by Expert BO * 



Target to Taachar ** I 

Target to Other Adult 2 

Target to Student 3 

Target to Tchr 1 Orp 4 

Taaehar to Target H 5 

Other Adult to Target 6 

t'tudent to Target T 

Teacher to Group 8 

Other Adult to oroup 9 

Student to Group 10 

Nona of the above 11 

Marginal Total! 



313- 



. Table 161 

Oaocrlptlve Study spr ingN 197a, 

, Reliability on interact lont 
Obiervet tQ Expert 80 
• % r . School eoi erode 2 

week 0 Day * 

k OMP 



Category ehoten by Obtertver 40v 
1 8 3 4 5 6 T 8 9 10 11 
♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 



O 



Datel 06-25-79 
Program! REIP9S 
Data Fl lei RElDl 



Marginal 'Nominal 
Totert; * Aoreement 

t " r 

♦♦♦♦♦♦♦♦ 



^ralnil Cxt*niioh 
AOr««m«nt ©* K«pp« 



0 
0 
0 
0 
0 

1 

0 
0 

r 

0 
0 



0 
0 

1 

0 
0 
0 
1 
0 
0 
0 
4 



0 
0 
0 
0 
0 



1 

0 
0 
0 
0 

1 

0 
0 
0 
0 

1 



0 
0 
0 

1 
1 

0 
0 



0 
0 
0 
0 

1 

0 
0 
0 
2 



0 
0 
0 
0 

1 

2 
0 

4 

5 
0 

4 

16 



0 
0 
0 
0 
0 

6 
o 

r 0 

t o 

0 
0 



. 0 
0 
0 

> 

0 
0 

0 
0 

*< o 
0 



52 



♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 



♦♦♦♦♦♦♦ 

64 

♦♦♦♦♦♦♦♦ 



0 


♦ 1 

A 1 ♦ 

♦ s 0 ♦ 


0,000 


.500 


«»024 


0 




— — ^ 




0 


♦ / 4 

♦ i ♦ 


1,000 


.167 


1.000 




♦ o ♦ 


X 






0 








0 


♦ • 2 •» 


0.000 


.667 


' -.037 


1 


♦ 6 ♦ 


.167 


. .333 


t .146* 


• 

0 


: ry2 : 


.500 


,667 


.461 


4 


♦ 8 * 


.500 


.500 


,362 


2 


♦ 8 « 

♦ e < 


0.000 


0.000 k 


0.000 


0 








45 


♦ 56 « 


.604 


.929 


.464 



314 



Category Ma^olnal Agreement! #47© 
Cohen*t Kappa I «32B 



4 



Phase IV 
DescrtPt 



Table 162 

• IOE Evaluation Project 
iva Study * spr |ng 1978 



Ral labl 1 1 tv on interact loot 
Observer 20 Expert 60 
School 801 Grade, 2 
Week 0 Day * 
DMP 



/ 

Data! 06-25-79 
Program! REIP9S 
Data F| lai RCID1 



Category chosen Cati 
by twpert 80 * 



Target to Tearher 
Target to Other Adult 
Target to Student 
Target to Tchr 1 Orp 
Teacher to Target 
Other Adul t to Targe* 
Student to Target 
Teacher to Group 
Other Adult to ©roup 
Student to Group 
hone of the above 



Category chosen by Observer 30 

3 * . 5 6 7 8 * 



Marginal Nominal 
10 Totals Aoreemant 



light is 
Heroine | extension 
Agreement el Kappa 



♦♦♦♦♦♦♦♦♦< 

Marginal Totals ♦ 8 0 
♦ 



10 



67 



♦♦♦♦♦♦♦♦ 
♦♦♦♦♦♦♦♦ 



1 


♦ 
♦ 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 


1 ♦ 


I ♦ 


0.000 


• 800 


• •063 


2 


♦ 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 ♦ 


0 ♦ 








3 


♦ 
♦ 
♦ 


0 


0 


1 


0 


0 


0 


0 


0 


0 


0 


0 ♦ 


1 < 


1,000 


• 500 


1,0*0 

mmmmw 


4 


♦ 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 ♦ 


0 « 








5 


♦ 
♦ 


4 


0 


0 


0 


0 


0 


0 


0 


0 


0 


1 * 


a 4 


> 0.000 


0.000 


0«000 


6 


♦ 
♦ 


3 


0 


0 


0 


0 


0 


0 


0 


0 


0 


a « 


6 4 


> 0.000 


0*000 


0*000 


7 


♦ - 
♦ 


0 


0 


1 


0 


0 


0 


0 


0 


0 


0 


1 < 


a i 


► 0.000 


0,000 


0,000 


8 


♦ 
♦ 


0 


0 


0 


0 


0 


0 


0 


* 


0 


0 


4 « 


, 8 < 


► .800 


,800 


,oa 


9 


♦ 


1 


0 


0 


0 


0 


0 


0 


3 


0 


0 


* 4 


> 8 < 


► 0.000 


0,000 


0*000 


10 


♦ 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 < 


i 0 < 


► mmmmm 




mmmvw 


11 




0 


0 


0 


0 


0 


0 


0 


3 


0 


0 


S3 < 


► 56 


» ,8*8 


,83ft 


,738 



s 



335 



ERIC 



Category Merolnel Agreement! %M 
Cohen's Kappa 9 ,321 



316 



Table 163 \ 

Pheea Iv - IOC evaluation Projtct 
Detcrlptlve Study - spring 197S 

Reliability orv mtaractlona 
Ofciarver 30 Export 90 
School 801 Oreda 
Weak 0 Day * 
DHP 



Data) 06-2S~79 
Programl REIP95 
Data F| lal RCUDt 



Catagory chosen by obiarver 30 



Marginal 



Catagory chotan Catt 










* 


S 


6 


T 


6 


9 


10 


11 


Total* 


by Expart 90 


* 




1 


8 


3 
















* 








♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 




Targat to Taech^r 


1 


♦ 


0 


0 


o • 


0 


0 


0 


0 


0 


0 


0 


0 ♦ 


0 « 


Target to Other Adul t 


S 


♦ 
♦ 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 ♦ 


w 


Targat to Studant 


3 


♦ 


0 


0 


3 


1 


0 


0 


0 


0 


0 


0 


a « 


6 < 


Targat to TcHr a. orp 


A 


♦ 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 


o i 


0 < 


Taachar ' to Targat 


5 




0 


0 


0 


0 

* 


1 


0 


b 1 


1 


0 


0 


0 i 


. 2 < 


Other Adult to Target 


6 


♦ - 


0 


0 


0 


0 


0 


0 


0 


0 


0 


v «- 


0 ' i 


> . 0 < 


Studant to Target 


T 


♦ 
♦ 


0 


• 0 


2. 


0 


0 


0 


0 


* 0 


0 


0 


0 < 


► 2 


Teacher to oroup 


a 


♦ 


1 


0 


0 


0 


0 


0 


0 


e 


0 


0 


* < 


► !•> 


Other Adult to Group 


9 


♦ » 

♦ 


0 


0 


0 


0 


0 


0 ' 


0 


0 


0 


0 


0 * 


► 0 


Student to OrouP 


10 


♦ 
♦ 


V 

°\ 


0 




0 


0 


0 


0 


0 


0 


0 


0 


► 0 


None of the abova 


11 


♦ 


1 


• 0 


1 


1 


0 


0 


0 


3 


0 


0 




► SB 






♦ 

♦ ♦♦ 


♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 


♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 


»♦♦♦♦♦♦♦♦ 


Marginal Totait 


♦ 


2 


0 


6 


2 


1 


0 




12 


0 


0 


BB 


* 78 






♦ 

















317 



EMC 



catagory Marginal Agraamantl ^*i9 
Cohan Kappa I »B3T 



Nominal 
Agraamant 



UlQHttt 
Marginal Extaniion 
Agraamant o* Kappa 



0.000 



♦ BOO 

• 500 

0*000 
.615 

.591 



1.000 
0.000 
,600 

0.000 
,923 



1.000 



•*se 

,*9* 

o.ooo 

,5*5 
,«30 



313 



\ 
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Table 164 



Phafi IV - IOE Evaluation Project 
Descriptive Study - spring lfte 

Reliability on interaction! 
Observer *0 Expert 90 
School 601 Grade ^ 
Week 0 Day * 

OMP 



Oatel 06-85-79 
Programi RELP9S 
OatatFllei PELD1 



Category chosen Catt 
by Expert 90 * 



Marginal Total* 



Category chosen by Observer *0 
123*56769 10 11 
♦♦♦>♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 



Mere I na | 
Totals 

♦♦♦♦♦♦♦♦ 



1 0 8 0 2 0 213 0 0 55 
♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 



♦♦♦♦♦♦♦♦ 
T8 



Target to Teacher 


I ♦ 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 ♦ 


0 " < 




Target to other Adult 


9 ♦ 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 ♦ 


0 < 




Target to Student 


3 ♦ 


0 


0 


4 


0 


0 


0 


1 


0 


0 


0 


1 ♦ 


* < 


► .667 


Target to Tchr i orp 


4 ♦ 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 ♦ 


0 < 




Teacher to Target 


5 ♦ 


0 


0 


0 


• 


1 


0 


0 


1 


0 


0 


0 ♦ 


a .< 


► ,600 X 


Other Adul t to Target 


6 ♦ 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 * 


0 < 




Student to Target 


7 ♦ 


0 


0 


1 


0 


0 


0 


1 


0 


0 


o' 


0 ♦ 


t i 


► #500 


Teacher to oroup / 


e \ ' 


0 


0 


0 


0 


1 


0 


0 


10 


0 


0 


t ♦ 


13 < 


. ,769 


0th9r Adult to OrouP 


9 ♦ 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 ♦ 


0 < 


' ( \ 


Student to opoup 


10 ♦ 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 „ ♦ 


0 < 




None o* th« abovo 


11 ♦ 


1 


0 


0 


0 


0 


0 


0 


2 


0 


0 


58 ♦ 


35 * 


► ,9*5 



• Lightts 
Nominal Marginal Extension 
Agreement Agreement of Kappa 



0.000 

• 833 

ItOOO 

U000 
1 1»000 

ItOOO 



,*87 

\ .*«7 
,783 



,915 



319 



Oetegory Marginal Aoreemtntl #10* 
Cohen* • Keppei tTlT 



320 



Table 165 

Pheie IV • IOE Evaluat ion Project 
DttcrlPWvc Study * spring 19TR 



V 



R#l lability on Interactions 
Obaarvor 10 Export 90 
School aol Grada 5 
* WaaK 0 Day 5 

. OHP 



Oatal 06-25-T9 
Program! RIUP95 
Data Fliai RELD1 



Category cfcotan Cat* 
« by rxpart 90 * 



Marginal Totals 



Catagory chotan by obsarvar 10 
1 a 3 ' * 5 * 7 8 9 10 11 



Marginal 
Totals 

++++++++ 



Target to Teacher 


1 ♦ 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 4 


' 0 


Target to Other, Adult 


2 * 


0 


0 


0 


0 


0 


0 


0 




0 


0 


0 4 


0 


Target to Student 


3 ♦ 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0' 


1 

I 4 


1 


Target to Tchr t Orp 


4 ♦ 


0 


0 


0 


0 


0 


.0 


0 


0 


0 


0 


0 4 


0 


Teacher to Target 


S * 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 


1 4 


1 


Othtp Adult to Target 


6 ♦ 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 4 


0 


Student to Target 


7 ♦ 


0 


c 


1 


0 


0 


0 ' 


0 


0 


0 


0 


0 i 


> 1 


Teecher to Oroup 


6 ♦ 


0 


0 


• 0 


0 


0 


0 


0 


0 


0 


0 


0 4 


► 0 


Other Adult to group 


9 ♦ 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 i 


► 0 


Student to oroup 


10 ♦ 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 


1 < 


► 1 


None o« the above 


11 ♦ 


0 


0 


1 


0 


0 


0 


0 


0 


0 


0 


25 '. 


► 26 



321 



0020000000 28 

Category Marginal Agreement! »tte 
Qohenta Kappa I »!!• 



♦♦♦♦♦♦♦♦ 
30 



Uightia 

Nominal Marginal extension 
Agreement Agreement •» Keppe 



0t000 



OtOOO 



0.000 



OtOOO 
• 9*2 



,soo 



0..000 



OeOOO 



0*000 



322 



• •0T1 



0 t 000 



0.000 



0.000 
.429 



Table 166 

Phete IV - IOE Evaluation Project 
Descriptive Study - Spring 19TB 

Reliability on interactions 
Observer 20 Expert 90 
School SOI Grade 8 
week 0 Day 5 

OMP 



oetel 06-25-79 
Program! RELP9S 
Oata F| lei RELO) 



9 

ERIC 



Category ehoian Cat. 
by Expart 90 # 



Marginal Total! 



Cataoory choion by obtarvar 20 

1 a 3*5 ft 7 • 9 10 U 



Marginal 
Totals 



* UQfcttt 

• Nominal Marginal Extamion 
Agraaniant Agraamant of Kappa 



cTargat to Taachar 


1 ♦ 


0 


0 


0 


0 ' 


0 


0 


0 


0 


0 


ft 


0 ♦ 


0 4 




Yaroat to Othar Adult 


2 ♦ 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 ♦ 


0 4 




Target to Studant 


3 ♦ 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 


1 4 


1 4 


> 0(000 


'argat to Tehr 1 Orp 


* ♦ 


0 


0 


0 


0 


0 


0 


0 


o' 


0 


0 


0 4 


0 4 




Taachar to Tfrgat 


5 ♦ 


\ 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 4 


1 < 


. 0,000 


Othar Adult to Targat 


6 ♦ 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 4 


» 0 < 




Studant to Taroat 


T ♦ 




0 


1 


0 


0 


0 


0 


0 


0 


0 


0 4 


« 1 < 


► 0(000 


Taachar to Group 


6 ♦ 






0 


0 


0 


0 


0 ' 


0 


0 


0 


0 4 


> 0 < 




Othar Adult to Group 


9 ♦ 




f ] 


0 


0 


0 


0 


0 


0 


0 


0 


0 < 


► * 0 * 




Studant to Group 


10 ♦ 


0 


0 

• 


0 


0 


0 


0 


0 


0 


0 


0 


1 < 


> 1 • 


► 0*000 


Nona of tha abova 


11 ♦ 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 


26 < 


► ' 26 < 


> 1.000 



♦♦♦♦♦♦♦♦♦♦ ♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 
1 0 1 o 0 



26 



♦♦♦♦♦♦♦♦ 

30 



323 



oetegory Marginal Agreement I (321 
Cohen •« Kappa I «19l * 



0(000 



1(000 



o.ooo 



0(000 



0(000 
,929 



324 



• •034 



0*000 



0.000 



0.000 
I. 000 



H 



Table 167 



H 
On 



\ 



Catagorv ehoian Cett 
by depart 80 * 



Marginal Totals 



325 



Phata Iv - I0E Evaluation project 
Daacrlptiva study • SPHno 197S 

\ Ral lability on mtaractlona 

Obiarvar 30 Expart 60' 
School 001 Orada 5 
Weak 0 Day 8 

OMP 

- 

eataoory choaen by Obaarvar 30 
I 2 3 h 9 6 7 6 9 10 11 
♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 



Oatai 06-25*79 
program! RELP9S 
Oata F|lai RELD1 



Targat to Taachar 


1 ♦ 


0 


0 


» 

0 


0 


0 


0 


0 


0 


0 


0 


0 


Target to Other Adult 


2 ♦ 


0 


0 


0 


0 


0 


0 


0 


0 


0 


' 0 


0 


Targat to Studant 


3 ♦ 


0 


0 


0 


0 


0 " 


0 


0 


0 


0 


0 


0 


Target to Tehr i orp 


* ♦ 

* ■ 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 


Taachar to Toroat 


5 ♦ 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 


Othar Adult to Targat 


6 ♦ 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 


Studant to Targat 


7 ♦ 


0 


0 


0 


0 


0 


0 


0 


• 

0 


0 


" 0 


I 


Taachar to Oroup 


ft ♦ 


0 


0 


0 


0 


0 


0 


0 


2 


0 


0 


1 


Othar Adul t to oroup 


9 ♦ 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 


Studant, to Oroup 


10 ♦ 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 


Nona o* tha abova 


il ♦ 


0 • 


0 


0 


0 


o . 


0 


0 


0 


0 


1 


2S 

1 



Ll0ht«a 

Marginal Nominal Marginal Extortion 
Tota | Agraamant A9raamant *f Kappa 



♦♦♦♦♦♦♦♦ 
0 



♦♦♦^♦♦♦♦♦^4 44* ♦♦♦♦♦♦♦♦♦♦♦♦ 

0 0 0 0 0 0 0 *0 1 •? 
♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦ ♦♦♦♦♦^♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦4 

C«taoorv Marginal Aortawtnti ( *0T 
Cohan it Kftpaat ,S3i 



0 
0 
0 
0 
0 
1 
3 
0 
0 

• 26 
♦♦♦♦♦♦♦♦ 
30 

♦♦♦♦♦♦♦♦ 



.....o • 

OtOOO j* 0 
,667 



#000 
tft«t 



OtOOO 
♦ 4*3 



0«000 
,963 



,•18 
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Table 168 

Phese IV • IOE Evaluation Project 
Descriptive study ? SPMre J?79 

Reliability on interact Ions , 

/Observer 40 Expert BO 
School 801 > Orede 5 
Week 0 Dey S 
OMP 



s 



Detel 06-25-79 
Programl REUP^S 
Data Fllei PElDl 



Category chotftn 
by Expert Ho 



I 

Cett 

7~r~ 



T 



Cetegory chosen, by Observer 40 

.. . 0 9 



io U 



Marginal 
Tatats 



Lightts 

Nom I na I Mere I na | Extent | on 
AWreeme*t Agreement *ap*#~ 



0 v 0 
0 0 



♦ 

♦ 
♦ 

♦ 



Target to Teacher 


1 


♦ 




« 0 


0 


0 


0 


0 


0 . 


0 


0 


Tercet to Other Adult 






0 


0 


. 0 


0 


0 


0 


«. 




0 


Target to Student 


3 




0 


0 


0 


0 


0 


0 


0 


0 


0 


Target to Tchr 4 or> 






0 


0* 


0 


0 


0 


0 


0 


0 


0 


Teacher to Target 


5 


♦ 




0 


*0 


0 


0 


0 


0 


0 


0 


Other Adult to Target 


6 




0 


0 


0 


0 


0 


0 


0 


0 


0 


Student to Target 


7 


♦ 


0 


0 


0' 

{ 


0 


0 


0 


0 


0 


0 


Teacher to Group 


6 




0 


0 


0 


0 


0 


0 


0 


3 


0 


Other Adult to orpup 


9 


« 


0 


0 


0 


0 


0 


0 


0 


0 


0 


Student to group 


10 


♦ 


0 


0 


0 


0 


0 


0 


0 


0 


0 


None of the above 


11 


♦ 


0 


0 


1 


1 


0 


0 


0 


2 


0 



0 
0 
0 
0 
0 
0 
0 
0 
0 



0 
0 
0 
0 

1 

0 
0 
0 
22 



Marginal Totals ♦ 
♦ 



23 



♦ 



o 

0 
0 

0 

0 

0 

1 

3 

0 
26 



30 



0.000 
1.000 



,0*6 



A ... 

0,000 
0.000 

0.000 

•too 

t 

v 

'*««*^ 
• •08 



0.000 
1(000 



>9'1 



1 



o 

ERIC 



327 



/ 



Category Marginal Agreement! 
Cohen* • Meppei t*7T 



328 



03 



Table 169 

Phase IV - I0E Evaluation Project 
Descriptive study - spring 1978 

Pal lability on interactions 
Observer io Expert 90 
School 801 Grade 2 
week o Day 8 
WDRSO 



DOtel 06-25»T9 
Progreml REI.P9S 
Data F I I • I RClOi 



Category eHoien Cat* 
by Expert 90 * 



Category chosen by Observer 10 

3 4 5 6 T 8 



9 10 



Taroet to Teacher 

Target to Other Adult 2 ♦ 

♦ 

Targets to student 



Tergtt to Tchr i Orp 
Teacher to Target 



Teeeher to oroup 
Other Adult to orouP 
Student to orouo 
None of the above 




3 ♦ 
♦ 

4 



Other Adult to Target 6 ♦ 
Student to Target 



T ♦ 
♦ 

a ♦ 
♦ 

9 ♦ 
♦ 

10 ♦ 

11 ♦ 
♦ 



Mergmel Toteia ♦ 
♦ 



0 
0 
0 
0 
0 
0 
0 
0 
0 
0 
0 



11 



0 
0 
0 
0 

1 

0 
0 
3 
0 
0 
36 



11 0 o *o 
♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 



category Marginal Agreemonti 
Cohen's xeppei t Slt 



Marginal 
Total e 

0 
0 
0 
0 
1 
0 
0 
8 
0 
0 
48 

♦♦♦♦♦♦♦♦ 
54 

♦♦♦♦♦♦♦♦ 



ligHtte 

Nominal Marginal extent |on 
Agreement Agreement of Keppe 



0,000 



.ess 



f aoo 



Of 000 
ItOOO 

o 6 ooo 

• T2T 

♦ 889 



*#019 



.529 



,289 



330 



fable 170 



Phata IV - IOC Evaluetldn Project 
Otter lot Ive Study - Spring 1976 

Re I lability on Interact font 
Observer 20 Expert 90 
School 601 Or ado 2 v , 
Waak 0 Day 5 

WDRSD 



Dotal 06*25-79 
Programi RELP9S 
^ Data *| !•! RttOl 



Category chosen Cat. 
b> Expert 90 ■ * 



Category chosen by Observer 20 

3 k 5 ' 6 T a ! 



10 11 



Marginal Totals 



^♦♦♦♦♦♦♦♦^♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦^♦♦♦♦♦♦♦♦♦♦♦♦♦t 

* * i 



Marginal 
Totals 

♦♦♦♦♦♦♦♦ 



Target to Teacher 


1 ♦ 


0 


0 


0 


0 


0 


0 


0 


> o 


0 


0 


' 0 4 


0 


Target to Other Adult 


2 ♦ 


0 


0 


0 


0 


,0 


0 


0 


0 


0 


0 


0 4 


0 


Target to Student 


3 ♦ 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 4 


0 


Target to Tchr I orp , 


4 ♦ 


0 


0 


0 


0 


0 


0 


0 . 

1 


0 


0 


0 


0 4 


0 


Teacher to Target 


6 ♦ 


' 0 


0 


0 


0 


0 


0 


0 


0 


0 


0 


1 ♦ 


1 


Other Adult to Target 


6 ♦ 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 


* 0 4 


0 


Student to Target 


7 ♦ 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 4 


0 


Teacher to Group 


a ♦ 


0 


0 


0 


0 


0 


0 


0 


1 


0 


0 


T* 4 


• 


Other Adul t to orouP 


9 ♦ 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 4 


0 


Student to oroup 


10 ♦ 


0 


'0 


0 


0 


0 


0 


0 


0 


0 


0 


0 4 


0 


None o# the above 


11 ♦ 


1 


0 


0 


0 


0 


0 


0 


0 


0 


0 


** 4 


*5 



+ 4 ♦♦♦♦♦♦ ♦♦♦♦♦♦♦♦♦♦♦♦♦ ♦♦♦♦♦♦♦♦♦ *♦♦♦♦♦♦♦♦♦->♦♦♦♦•> ♦ + 

1 0 0 0 0 0 0 1,0 0 82 



♦♦♦♦♦♦♦♦ 
♦♦♦♦♦♦♦♦ 



Nominal 
Agreement 



Light's 
Marginal Extension 
Agreement of Kappa 



0t000 



,185 



\ 



O«00Q 



0,000 



0,000 



loo 



,9*8 



• 400 



331 



cataoory Marginal AgraaMntl 
OoHan't Mppai ,l*a. 



332 



. Table 171 

Phase IV • IOE Evaluation Project 
Oaserlptlve study - sprang 19T6 

Reliability on Intaractloni 
Obieryer 30 Expert. GO . 
School 601 Orada 2 
Waak 0 Day 5 

WDRSD 



ro 

O 



Data) 06-25«79 
Programl REIP9S 
Data Fliai RElOl 



Cataoory chosen < Catt 
by depart SO # 



Marginal Totals 



Cataoory chosen by Observer 30 
1 2 3 * 5 6 7 6 9 10 11 



Marginal 
Totals 

♦♦♦♦♦♦♦♦ 



ugfctit 

Nominal Marginal Octant | an 
Agreement AOraement o# Kappa 



333 



I 

I P 3 1 10 * * 0 0 AO 
♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦*♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 

Cataoory Mergfnel Agreemanti *B30 
Cohen »• Kappa I «3U 



♦♦♦♦♦♦♦♦ 
SA 

«-♦♦♦♦♦♦♦ 



9 

ERIC 



Target to Taachar 


I ♦ 


0 


, o 


♦ 0 


0 


0 


0 


0 


0 


g 


0 


0 < 


► 0 < 


r 

> 4* — * mm- 4 


0*000 


*«»■»•** 


Target to other Adult 


2 ♦ 


0 


0 


0 


0 


0 


0 


o 


0 


0 


0 


0 < 


► 0 < 








Target to ?'<uo«nt 


3 ♦ 


0 


0 


'1 


0 


0 


0 


o 


0 


Ov. 

0. 


0 


3 < 


> A 4 

► .0 4 


► ,250 


*T50 
0*000 


*206 


Target to Tchr 1 orp 


A ♦ 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 < 






Taachar to Target . 


S ♦ 


0 


0 


0 


0 


0 


0 


0 


0 


. 0 

e< 


0 


1 < 


> 1 < 


► 0,000 


1*000 


• *019 


Other Adult Target 


6 ♦ 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 « 


► 0 4 








Student to Target. 


T * 


0 

1 


0 


0 


0 


0 


0 


I 


0 


0 


0 


0 < 


« 2 4 


> 1*000 


1*000 
0*000 


1*000 


Teacher to group 


a ♦., 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 4 

' 0> < 


► 0 < 




>' 


Other Adul t to group 


9 ♦ 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 


► 0 < 






Student to group 


10 ♦ 


o 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 i 


k 0 < 








None o# t|4 ebove 


11 ♦ 


1 


0 


a 




1 


0 


0 


1 


0 


0 


Al < 


► AT 4 


> ,gT« 


,9BT 


,I3A 



334 



/ 



Category ehoien Cat# 
bv EKPart 60 # 



Target to Teeeher I 
Target to Other Adult 2 



Target to Student 
Turget to Tchr l opp 
Toacher to Target 
Other Adult to Target 
Student to target 
Tuacher to oroup f 
Othor Adult to oroup 
student to oroup 
None of the above 



3 
4 
5 
6 
7 

e 

9 
10 
11 



Marginal Totals 



♦ 
♦ 

♦ 
♦ 
♦ 

♦ 

♦ 
♦ 

♦ 

♦ 

♦ 
♦ 



0 
0 
0 
0 
0 
0 
0 
0 
0 
0 
0 



Table 172 

Phase XV - !QE Evaluation Project 
DatcrlPtlve Study - sprjnfl 1»T8 

Reliability on interaction* 
Obtervar 40 Expert 80 
School 801 orada 2 
Week 0 Day 5 

WORSO 



Category chonen by Obeerver 40 



10 11 



46 



Marginal 
Totals 

♦♦♦♦♦♦♦♦ 



0 


0 


0 


0 


0 


0 


0 


0 


0 


0 ♦ 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 ♦ 


0 


0 


3 


1 


0 


0 0 


0 


o 


0 


0 


0 ♦ 





0 


0 


0 


0 


0 


0 


0 


0 


o " 


0 ♦ 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 


I i 


1 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 4 


0 


0 


1 


1 


0 


0 


0 


0 


0 


0 


0 4 


> 2 


0 


0 


0 


0 


0 


0 


0 


0 


0 


9 < 


> 0 


0 


0 


0 


0 




0 


0 


0 


0 


0 i 


► 0 


0 


0 


0 






0 


0 


0 


0 


0 i 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 


4? < 


► 4T 



♦♦♦♦♦♦♦♦ 
8* 



Oatal 06»25«7» 
programs REL.P9S 
Data Fllal REID1 



Nominal Marginal extant ion 
Agraamant aflraamant Kappa 



1 760 



0*000 



0*000 



1*000 



1*000 
0*000 
0*000 

0*000 



,»7» 



,730 



Qi/000 



0*000 



1*000 



335 



Catagor* Marginal aoraamanti .J«» 
Cohan* • Kappa* 



336 



N3 



Table 173 

Phase IV - IQC Evaluation Project 
Descriptive Study • Spring 19TS 

Reliability on interact Iqpa 
Observer 10 Expert SO' 
School 801 Grade 2 
Week 0 Day 5 
OMP 



Data! 06«25»79 
Progrewl REUP9S 
Data Fllel RELD1 



Category chotan Cat. 
by Expert 00 *■ 



Cateoory chosen by Observer 10 
3 4 5 6 T a 9 



10 11 



Marginal 
Totalt 



Targat to Teach*? 


♦ 

1 ♦ 


2 


0 - 


0 


0 




0 


0 


0 


0 


0 


0 


Target to Othar Adult 


♦ 

8 ♦ 


0 


0 


0 


0 < 


* 0 


0 


0 


0 


0 


0 


0 


Tar eat to Student 


3 ♦ 
♦ 


0 


o 


2 


0 


0 


0 


0 


0 

» 


0 


0 


0 


Targat to Tchr V Orp 


A ♦ 


0 


0 


0 


0 


0" 


0 


0 


0 




0 


0 


Teacher to Target 


♦ 

5 ♦ 


0 


0 


0 


0 


0 


0 


0 


0 


1 


0 


1 


Other Adult to Target 


♦ 

6 ♦ 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 


Studant to Target 


T ♦ 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 


Teacher to oroup 


♦ 

a ♦ 


0 


0 


0 


1 


0 


0 


0 


9 


0 


0 


11 


Other Adult to Group 


9 ♦ 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 


Studant to orouP 


10 ♦ 


0 


1 


0 


0 


0 


0 


0 


0 


0 


0 


1 


Nona o# the above 


♦ 

11 ♦ 


1 


0 


0 


0 


0 


0 


0 


A 


0 


0 


AA 



Nominal 
Agreement 



1*000 



UQfct'9 
Marginal Extension 
Agreement Keppe 



4 

♦ 

Marginal Total) ♦ 3 I 



21 000 13 1 0 ST 



337 



Category Marginal Agreement} ,*A» 
Cohan Kappai ,ASB 



1,000 
0,000 

..... 

,A29 
0,000 

,e«t 



,66T 
0,000 
1,000 
0,000 
0,000 



,61* 

0,000 
0,000 

• •to 



338 



1.000 
1,000 
0,000 

• 31A 

0,000 
,ell \ 



.)■' 



Table 174 

Phata IV • IOE Evaluation Pfojact 
Otacrfp||va study •sprmO 19T8 

Pal lability on mtaractlom 
Obtarvar 20 Export 80 

School 601 Orada 2 
* WaaN 0 Day 5 
OMP 



Oatal 06-2S-T9 
Prooraml PCi.P9S 
Data PI lai RELDi 



Cataoory ehotan Catt 
by Export SO * 



Marginal Totals 



Cataoory ehotan by Obaarvtr 20 
\ 2 9 4 5 6 T 8 9 10 11 



Taroat to taachar 


1 ♦ 


2 


0 


0 


0 


0 


0 


0 


0 


0 


Taroat to Othar Adult 


2 ♦ 


0 


0 


0 


o , 


0 


0 


0 


0 


0 


Taroat to st.udant 


3 ♦ 


0 


0 


2 


0 


0 


0 


0 


0 


0 


Tarqat to TcKr I Orp 


* ♦ 


0 


0 


0 


0 


0 


0 


0 


0 


0 


Taachar to Taroat 


5 ♦ 


0 


0 . 


0 


0 


1 


0 


0 


0 


0 


Qthti* Adult to Targat 


$ ♦ 


0 


0 


0. 


0 


0 




0 


0 


0 


Studant to Taroat 


T ♦ 


0 


0 


0 


0 


0 


0 


0 


0 


0 


Taachar to oroup 


8 ♦ 


1 


0 


0 


0 


1 


0 


0 


IT 


0 


Othar Adult to oroup 


O ♦ 


0 


0 


0 


0 


0 


0 


0 


0 


0 


Studant to oroup 


10 ♦ 


0 


0 


0 


0 


0 


0 


0 


1 


0 


Nona of tha abova 


11 ♦ 


0 


0 


0 


0 


0 


2 


0 


6 


2 



3 0 2 



0 
0 
0 
0 
0 
0 
0 
0 
0 
1 
0 



2* 



0 
0 
0 
0 
1 
0 
0 
2 
0 
0 
39 

*2 



Marolnai 
Totals 

«««««««« 
2 

0 

2 w 
0 
2 
0 

9 / 
21 
0 
2 
♦9 
««««««« 

Ti 

♦♦♦♦♦♦♦♦ 



Nominal 
MraemaHt 



1,000 



UOM'I 
Marginal Cxtaniion 
Aoroamant of Kappa 



1,000 
,500 

,aio 

,500 
,79* 



,667 

« 

1,000 

1,000 
0,000 

,678 
0,000 
,500 
,557 



1,000 
1,000 
,467 

,725 

,*9* 
,558 



339 



Catoeory Marginal Aoraamontl 9*11 
Cohantt K«>pai 



340 



N> 



Table 175 ' 

Phete IV • IOE Evaluation Project 

DetcMPtlve study • SPHng 19TO 

Reliability on interactions 

jdtervtr 30 Expert 90 

School 601 Orada S 

Waej^lT\ Oay 8 
D* 

\ ♦ 



4S 



Oattl 0*«25-79 
Program t REIP9S 
Data Fllol REkDl 



Catagorv choaen Cat* 
by Expert 90 . * 



Marginal Total t 



341 



Catagorv ehoaan by Obtaryar 30 4 \ 
1 2 3 A 5 6 T \8 9 10 U 



\Margl ne| 
totals 



light ft 

Nominal Marginal Extant) on 
Agreement AQraement o# Kappa 
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Abstract 



This project paper reports the interobserver agreements and 
reliabilities for the observation procedures used in the Descriptive 
Study of Phase IV of the IGE Evaluation Project. Only data from 
observers at the two DMP and two WDRSD schools are reported in th^s 
paper. The four observers attended a four day training session, 
January 10-13 1978, to learn how to use the observation procedures. 
On March 16 and 17 a retraining session was held at the middle of the 
fourteen week investigative period to check on the interobserver 
agreements. In addition, on-site visits were made to each school by 
one of the Phase IV staff where both the observer and staff member did 
one or two observations together. Interobserver agreement and reliabil- 
ities were computed for each of these three, situations — training 
session, retraining session, and on-site visits. For each pair of 
observers and for each category, marginal agreements were computed. For 
the sets of categories— General Content, Pacing, Grouping, Materials, 
Learner Moves and Interactions — Category Marginal Agreements and 
Cohen's Kappa were computed. Generalizability coefficients for estimates 
of reliability were computed for the training and retraining sessions 
for groups of three observers observing two or three classes for two 
days. * The results indicated that there was high interobserver agreement 
at the three differenjt check t^mes on the categories of self -paced, other 
paced, individual, large group, paper /pencil, engaged-written, engaged- 
covert, and non-engaged off task. The low interobserver agreements 
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were, explained mainly as a result of a low frequency of occurrence. 
The generalizability coefficients for total engagement and total 
non-engagement were acceptable. 
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OaicMfrtlva Study • Spring im 

p«| lability on Oanaral Contant 
Obiarvsr /& Export 9G 
School BOH Orada 8 
wteK 9 Dav h 
DMP 



Catagory chotan 
by Export to 
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Trani i t ion 
^anagamant 
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Othar AcadarMc 
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♦ 
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♦ 
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♦ 
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♦ 

T ♦ 
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0 
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0 
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8 
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pheta IV - IGE Evaluation Project 
Descriptive Study ~ Spring 1978 

Rt M ability on Genera* Content 
Obterver 20 Expert 90 
School 801 Grade 8 
» Week 9 Day 4 

OMP 



Category choeen 
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Transition 
penegement 
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♦ 
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♦ 
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♦ 
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0 
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♦ 
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0 


0 


0 


0 


0 


0 
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0 


0 


0 


0 


0 


0 
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♦ 
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0 
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0 
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Pheia IV • IOC Evaluation Project 
Descriptive study - spring 1978 

Pal lability on General Content 
Observer ^0 expert 00 
School eoT Grade 3 
week 9 Day * 
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♦ 
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♦ 
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♦ 
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♦ 
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♦ 
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0 
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0 
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0 
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0 
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0 
0 
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0 
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0 
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0 
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1 
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♦ 
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♦ 
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4 
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pheie IV - IGE Evaluation projact 
DatcrlPtlva Study » soring 1978 

Pal lability on General Content 
Observer 10 Expert 90 
School sol Qrada S 
week 9 Day 5 

OHP 



Category choten 
by Expert 90 
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Management 
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Nonoeadem ( c 
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Nion* o# tha Ae>ove 
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♦ 
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♦ 
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0 
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0 
5 
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0 
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Phaie tv - IOC Evaluation Project 
OatcrlPtlva Study - SPrino 19T6 

pal lability on General Content 
Obiarvar 20 Expert *0 
school 801 Oracle S 
weak 9 Day § 
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by E*P*rt 90 * 
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Nona of the Above t 

Marginal Total* 



♦ 
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♦ 
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♦ 
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♦ 
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Phalli IV - IGC Evaluation Project 
Descriptive Study • Spring 1978 

Reliability on General Content 
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School SOI Grade B 
week t Day 5 

OHP , 
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Melt 1 
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Nonecedemjc S 
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0 
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0 
0 



0 
0 
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0 



0 
0 
0 
0 
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0 
0 
0 
0 
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I 

0 
0 
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0 

1 
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0 

1 
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06 
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Ph.M IV - XOE cvtlujitlon>rojtct 
OaocrlPtlva study - sprinq Wi 

Pliability on oanaral Contont 
tobtarvar *o expart 80 
School 801 Orad* S 
waafc 9 Day 3 
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Cataoory choion 
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Transition 
Kanagamant 
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0 

0 
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♦ 
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♦ 
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♦ 
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♦ 
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0 
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0 
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phata IV. • IOE Evaluation Projact 
DaicrlPtlve Study - spring 19T8 

Ml lability on oanarai Contont 
Obiarvar 10 Export 90 
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WDftSD 
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♦ 
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♦ 



Catagory choaon by Obaorvar 10 
1 2 3 * 3 6 T 

ooooo 
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♦ 
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♦ 
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Ph«$a IV - IOC evaluation Project 
DaicMPtlva study • spring 1*78 

Pal lability on Otnaral Contant 
Obtarvgr 20 K*P«rt 90 
SChOOl 601 Grada 2 
tfaak 9 Day 5 
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by c«P«^t 10 
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♦ 
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♦ 
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0 
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♦ 
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0 


0 
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phata IV - IOt Evaluation Project 
DaieriPtiva Study • SPrino 197S 

^al lability on Oaneral Content 
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♦ 
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♦ 
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0 
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0 



0 
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0 
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!»Hait IV • IOE Evaluation Projact 
OticrlPtlvt Study • 3Prin« 19T6 

pal lability on oanaral Contant 
Obtarvar *o Exoart SO 
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Weak 9 Day 9 
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Mai t 
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♦ 
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♦ 

♦ 
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♦ 

♦ 
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♦ 
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♦ 
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0 



0 
2 

0 
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0 
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0 
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Phait iv - IOC Evaluation Projact 
OatcMPtlva Study - SPrlno 1979 

Pal lability on oanaral contant 
Obtarvar 10 Fxpart 90 
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Waak 9 Day 5 
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Phaaa IV - XOf evaluation Projact 
OatcriPtlva Study - Spring 19TB 

Ral lability on Oanaral Contant 
Obaarvar 20 Expert 90 
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waak 9 Day 5 
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Phase IV - IOC Evaluation Projact 
DatcriPtlvtt Study « soring 
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0 
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Phaia XV » IOC Evaluation Projact 
OatcriPtlva Study • Spring 1976 
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Phase iv - IGE Evaluation Project 
Descriptive Study •> Sprlna 1978 
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School 601 Grade 5 
Week 9 Day k 

. DMP r 



O 



Category chosen 
by Expert 90 



Cat , 



Category chosen 
by Observer 10 



1 



Marg I na 
Total s 



Nom | na l 
Agreement 



Marg I na I 
Agreement 



L Ight « s 
Extens I on 
of Kappa 



Sel f 

Othf.r 
Ne I ther 



1 



Mara | n« 



♦ 
♦ 
♦ 

2 + 
♦ 

3 ♦ 
♦ 



Toto i s ♦ 



76 
2 
1 

61 



4 
1 

0 



0 
0 



♦ 

♦ 
■f 
♦ 

+ 



82 
3 
5 

90 



♦ 
♦ 

♦ 

♦ 

♦ 
+ 
♦ 



.951 
,333 
• 800 



• 988 

• 600 
.800 



• 512 
.29* 
.791 



' 1 *•* 



category Marginal Agreement I ,796 
Cohan's Kapnai ,5^7 



ERIC 



Table 1.99 

Phase IV - IGE Evaluation Project 
Descriptive Study - spring 1978 

Re I I ab I I I ty on Pace 
Observer 20 Expert 90 
School 601 Grade 5 
Week 9 Day 4 

v DMP 



Category chosen 
by Expert 90 



Cat . 



Category chosen 
by Observer 20 



Mara I n« 
Tota I « 



Nom | na I 
Agreement 



Maro I na I , 



t lght»i 
Extens | on 



Agreement of Kappa 



se I f 
Other 
Ne I ther 



1 



♦ 
♦ 

2 ♦ 
♦ 

3 ♦ 



80 
3 

0 



0 
0 
0 



Mar g I na I 



Totals 83 0 

♦ 

♦♦♦♦♦♦♦♦ 



2 
0 
5 



♦ 
♦ 
♦ 

♦ 
♦ 
♦ 

♦ 

♦ 



82 
3 
5 

90 



♦ 
♦ 

♦ 

♦ 

♦ 

■ ♦ 



a 976 
0.000 
1*000 



.988 
0.000 
• Tie 



,686 
0.000 
1.000 



category Marginal Agreement! .567 
Cohen's Kappal ,643 



er|c 



Table 200 

Phase IV - IGE Evaluation Project 
Descriptive Study * spring 1978 

Re I I ab I It ty on Pace 
Observer 40 Expert 8o 
School 601 Grade S 
Week 9 Day 4 
DMP 



Category chosen 
by Expert 80 



Sel * 
Other 
Ne I ther 



Category cbosen 
by Observer 10 



Cat « 
n 



1 ♦ 
♦ 

3 ♦ 



♦ ♦♦♦♦♦ * ♦ ♦ 4 
♦ 

27 



4 
2 



2 
45 
4 



0 
0 
6 



MarO I na I 
Total s 



♦ 
♦ 
♦ 
♦ 
♦ 
♦ 
♦ 



29 
49 
12 



♦ 
♦ 
♦ 
♦ 

♦ 



Nom | na I 
Agreement 



.931 
.918 
.500 



Marg | na I 
Agreement 



.879 
.961 
.500 



Light is 
Extens | on 
of Kappa 



.891 



• 812 
.464 



Marginal Totals 



33 51 



♦ 
♦ 



90 



♦ 
♦ 



category Marginal Agreement! 
Cohan's Kappal .764 



.780 



'I'ab U- 201 



Phase Iv - IGE Evaluation Project 
Descriptive Study - spring 1978 

Re I I ab I I I ty on Pace 
Observer 10 Expert 90 
Schoo 1801 Grade 2 
Week 9 Day h 
WDRSD 



tegory chosen 
by Expert 90 



Sel f 
Other 
Ne I ther 



Cat « 



Marginal Totals 



♦ 

1 ♦ 

2 ♦ 

3 ♦ 
♦ 

♦ ■ 



Cateqory chosen 
by Observer 10 



1 



6 
0 
1 



2 
3 

51 



0 
0 
8 

8 



Maro I na I 
Total s 



♦ 
♦ 
♦ 
♦ 

+ 
♦ 
♦ 



6 
46 
12 

66 



♦ 

♦ 

■f 
♦ 

♦ 

f 



Nom | na I 
Agreement 



• 75 0 
1.000 
,667 



Marg I na I 
Agreement 



Light t s 
Extens I on 
o* Kappa 



,875 / ,720 
,902 1,000 
,667 ,621 



9 



category Marginal Agreement! ,81«5 
Cohen's Kappal ,787 



to 
U" 
00 



3 85 



Table 20 2 



Phase TV - IOE Evaluation Project 
Descriptive Study * SPri.no 1978 

Re I lab I.I I ty on Pace 
Observer 20 Expert 9o 
School 801 Oracle 2 
Week 9 Day 4 
WDRSD 



Category chosen 
by Expert 90 



Cat, 
ft 



Category chosen 
by Observer 20 



1 



Maro I na 
Total s 



Nom j na I 
Agreement 



Marg I na I 
Agreement 



Llght«s 
Extens I on 
of Kappa 















♦ 




♦ 








Sel f 


1 


♦ 


6 


1 


1 


♦ 


8 




,750 


1,000 


,716 
















♦ 








Other 


2 


♦ 


2 


44 


0 


♦ 


46 


♦ 


,957 


,958 


,841 






♦ 








♦ 












Ne I ther 


3 




0 


3 


9 




12 


♦ 


,750 


,833 


,705 






♦ 








♦ 
















♦ 








♦ 




♦ 








Maro 1 na | 


Tota 1 s 




8 


48 


10 




66 


♦ 






* 






♦ 








♦ 




♦ 









category Marginal Agreementl ,931 
Cohen's Kappal ,765 



3b7 



■ 4 



Table 203 

Phase IV - IGE Evaluation Project 
Descriptive Study - Spring 1978 



Re I I ab I, I I ty on Pace 
Observer 40 * Expert 80 
School 801 Grade 2 
Week 9 Day 4 
WDRSD 



\ 



Category chosen 
by Expert 80 



Cat # 
H 



Category chosen 
by Observer M0 



1 



Marg I na I 
Tota I s 



Nom | na I 
Agreement 



Marginal 
Agreement 



Llghtts 
Extens | on 
of Kappa 







♦ . 








4 




4 








Sel f 


1 


♦ 


5 


0 


0 


4 


5 


4 


1 .000 


.556 


1,000 






♦ 








4 




4 








bther 


2 


♦ 




33 


0 


4 


37 


4 


• 892 


.892 


.760 






♦ 








4 




4 








Net ther 


3 


♦ 


0 


0 


18 


4 


18 


♦ 


1*000 


1.000 


1,000 






4 








4 




4 












*44 
♦ 








4 




4 








Marg 1 na i 


Total s 


4 


9 


33 


18 


4 


60 


4 












4 








4 




4 









category Marginal Agreement) ,816 
Cohen's Kappal ,881 



9 

ERIC 



38b 



38<J 



V 



Table 204 

r V 

Phase IV - IG€ Evaluation Project 
Descriptive Study - spring 1970 



a- 



Re I lab I I 1 1 v on Pace 
Observer \0 Expert 90 
School 001 Grade 2 
Week 9 Day 4 
DMP 



V 



Category chosen 
by Expert 90 



Cat ♦ 



Category chosen 
by Observer AO 

'l \- 2 3 * 



Marg I na I 
Total s 



Nom | na I 
Agreement 



Marg I na I 
Agreement 



Light's 
Extens I on 
of Kappa 



Sel f 
Other 
Ne I ther 



\ 



1 ♦ 
♦ 

2 ♦ 

3 ♦ 



3 

0 



5 
4 



1 

0 



0 J 1 



♦ 



70 
7 
7 



♦ 



9H 



.571 



1.000 



• 957 

• 778 

• e75 



,576 
.520 
1,000 



Marginal Totals ♦ 



67 



8 



64 



4 



390 



Category Marginal Agreementl ,870 



Cohen's Kappal ,664 



391 



9 

ERIC 



X 



Table 205 

Phase ly - IGE Eva I uat I on Project 
Descriptive Study - spring 1978 

Re I I ab I I I ty on Pace 
Observer 20 Expert -9o 
Schoo.l 801 Grode 2 
Week ' 9 Day 4 
•DMP 



Category chosen 
by Expert 90 



Cat. 
n 



Cateaory chosen 
by Observer 20 



Mara I na 
Tota I s 



Norn | na I 
Agreement 



Marg I na I 
Agreement 



. Light »s 
Extens | on 
of Kappa 



Sel f 



^ *» U *S am 



Nel ther 





♦ 




















i 




69 


0 


1 




70 




,986 


,959 


,891 




♦ 




















2 




4 


3 


V 




MM 

f 




,425 


,429 


,407 
























3 




6 


0 


7 




7 




1,000 


,875 


1 ,000 

























Marginal Totals ♦ 



73 



8 



♦ 
♦ 



84 



♦ 
♦ 
♦ 



Category Marginal Agreement! ,754 
Coh«n»s Kappat ,775 



ERIC 



332 



393 



Table 206 ' 

Phase iv - I GE Evaluation Project 
Descriptive Study •. SPr|nai978 . 

Reliability on , Pace 
Observer ^0 \ Exp"ert 80* 
School 801 Grade 2 
Week .9 Day 4 
DMP 



to 

or 



v Category chosen 
by Expert 80 



Cat; 



Category chosen 
by Observer 10 



1 



Marg I na 
Totals 



Nom | na | 
Agreement 



Mar g I na I 
Agreement 



Light's 
Extens I on 
of Kappa 



Se 1 f 


1 




43 


• 1 


0 


♦ ' 


44 ♦ 


.977 


t A4ft 




.932 


Other 


2 




8 


22 


0 


♦ 


30 , ♦ 


#733 


.833 




.608 


Ne 1 ther 


3 




1 


2 


1 




4 ♦ 


a .250 


.250 




,240 



























Marginal Totals ♦ 52 25 l ♦ 

♦ ♦ ♦ *• ♦ ♦ ♦ 



78 



Category Marginal Agree/entl .643 
Cohen's Kappai ,692 



395 



Table 208 

Phase IV - JGE Evaluation Project 
Descriptive Study - spring 1978 

Reliability on Pace 
Observer 10 Expert 90 
School 801 Grade 5 
week 9 Day 5 
OMP # 



Category chosen 
by Expert 90 



Se l # 
Cther 
Ne I ther 



Cat, 
n 



♦ 

1 ♦ 

2 ♦ 

3 ♦ 



Marginal Totals 



♦ 



Cateqory chosen, 
by Observer 10 



T3 
2 
2 

77 



2 
0 
0 



0 
0 
11 

11 



Marg I na 
Tota I s 



♦ 

♦ 
♦ 
♦ 
♦ 

♦ 

♦ 
♦ 



75 
2 
13 

90 



Nominal Marginal 
Agreement Agreement 



♦ 
♦ 
♦ 

♦ 
♦ 
♦ 

♦ 
♦ 
♦ 



.973 
0,000 
,846 



.974 
1,000 
.846 



Llght»s 
Extens I on 
of Kappa 



,815 
• ,023 
,825 



Category Marginal Agreementl ,940 
Cohen's Kappal .752 



396 397 



. , TaBle 207 

Phase IV • IGE Evaluation Project 
Descriptive Stud/ - spring 1978 

Reliability on Pace 
Observer 20 Expert 90 
Scbool 801 Grade 5 
Week 9 Day 5 
DMP 



c 



Category chosen 
by Expert 90 



Ca I • 
* 



Category chosen 
by Observer 20 



1 



Marg I na 
Total s 



Norn | na l 
Agreement 



Marg | na I 
Agreement 



Light's 
Extens I on 
of Kappa 







♦ 








♦ 




♦ 








Self 


1 


♦ 


74 


1 


0 


♦ 


75 


+ 


#987 


• 974 


• 908 






♦ 




















Other 


2 


♦ 


2 


0 


0 


♦ 


2 




0*000 


• soo 


••on 






♦ 








♦ 












Ne 1 ther 


3 


♦ 


1 


0 


12 




13 




♦ 923 


• 983 


• 911 






♦ 








♦ 
















♦ 




















Marg 1 na 1 


tota l s 


♦ 


77 


1 


12 


♦ 


90 



































393 



Category Marginal Agreementl .799 
Cohen's Kappal ,834 



Table 209' 

Phase IV - IOE Evaluation Projaet 
Descriptive Study '« spring 1978 

Reliability on Pace 
Observer 30 Expert 80 
School 801 Grade 5 
Week 9 Day 5 
DMP 



Category chosen 
by Expert 80 



Self 
Other 
Ne I ther 



Cat, 
u 



1 ♦ 

2 * 

4 

3 «• 



Cateaory chosen 
by Observer 30 



1 



28 
0 



Marginal Totals ♦ 29 

♦ 



61 



Llght»s 









Marg 1 na 


I- 


Nom | na | 


Marg 1 na I 


Extens | on 


2 


3 




Total s 




Agreement 


Agreement 


of Kappa 






♦ 












10 


0 


♦ 


38 


♦ 


.737 


• 763 


,612 






♦ 




♦ 








50 


0 


♦ 


50 


♦ 


1*000 


,820 


1.000 






♦ 




♦ 








1 


0 




2 


♦ 


0,000 


0,000 


0,000 










♦ 









90 



+ 
+ 



Category Marginal Agreement! t52fl 
Cohen'* Kappal ,726 



400 



401 



Table 210 

Phage IV - IOE Evaluation Project 
Descriptive study - spring 1978 

Reliability on Pace 
Observer 40 Expert 80 
School 801 Orade 5 
Week 9 Day b 
DMP 



CT 



Category chosen 
by Expert 80 



Cat . 



Category choten 
by Observer AO 



Marg I na 
Tota I s 



Norn | na l 
Agreement 



Mar g | na I 
Agreement 



Llght»s 
Extens I on 
of Kappa 



Sel f 
Other 
Ne I ther 



+ 

1 ♦ 
♦ 

2 ♦ 

3 ♦ 
♦ 



3% 
5 

0 



3 
45 
0 



0 
0 
2 



♦ 
♦ 
♦ 
♦ 

♦ 

♦ 



38 
SO 
2 



.V 

♦ 
♦ 
♦ 



.9*1 
.900 
1.000 



.950 
,960 
1 .000 



.858 
.786 
1.000 



Marginal Totals ♦' 

♦ 



AO 



48 



♦ 

* 
♦ 



90 



♦ 
♦ 



4r , 0 Cohen's Keppal ,828 



Category Marginal Agreementl .970 



9 

ERIC 



V 



Table 211 

Phase IV - IOE Evaluation Project 

Descr|ptlve\ Study * spring 1978 

\ 

\ 

Re I l ab I I I ty on Pace 

Observer 10 Expert 9o 

Schqo I 801 Grade 2 

Week 9 Day .5 

WDRSD 



Category chosen 
by Expert 90 



Cat* 
U 



Category chosen 
by Observer 10 



Mara I na 
Total s 



Nom | na I 
Agreement 



Marg | na r 
Agreement 



U loht » s 
Extens | on 
of Kappa 







4 








♦ 




♦ 








self ' 


1 


4 


23 


1 


1 


♦ 


25 


♦ 


#920 


.781 


.845 






♦ 








♦ 




♦ 








Other 


2 




9 




0 


♦ 


33 


♦ 


.727 


.750 


.561 














4 




4 








Ne 1 ther 


3 




0 


0 


e 


♦ 


e 


4 


UOO0 


.689 


1 . 00.0 






♦ 








♦ 




♦ 












4 




















Marg | na 1 


Totfl 1 S 


4 


32 


25 


9 


♦ 


66 


4 












♦ 








♦ 













category Marginal Agreement! ,809 
Cohen's Kappal e 727 



404 



405 



.T^ble 212 

Phase IV - IOE Evaluation Project 
Descriptive Study*- spring 1978 

Reliability on Pace 
Observer 20 Expert 90 
School 801 Grade 2 
Week 9 Day 5 
WORSD 



Category chosen 
by Expert 90 



Cat , 



Cateaory chosen 
by Observer 20 

i Margina 
12 3 Totals 



Nom | na I 
Agreement 



Light »s 
Marginal Extension 
Agreement o* Kappa 









+ + + + + + +++ ++++++ + + +V+ + ++++.++ + ♦ 














♦ 




















Sal t 




I 


♦ 


23 


2 


0 




25 




,920 


• 735 


,635 






















( 


,*76 


Other 




2 




11 


22 


0 




33 




.667 


• 727 


























1,000 


Ne 1 ther 




3 


♦ 


0 


0 


8 




e 




1,000 


1 .000 
















♦ 






































Marg 1 na 1 


Total « 




34 


24 


8 




66 





































40G 



Category Marginal Agreementl ,821 
Cohan » s Kappa l »676 



407 



Table 213 



Phase IV - IGE Evaluation Project 
Descrfptlve Study - spring 1978 

Re I lab I I I ty v pn Pace 
Observer 30 Expert 8q 
School 801 Grade 2 
Week 9 Day 5 
WORSD 



Category chosen 
by ExPer.t 80 



Cat 
u 



Category chosen 
by Observer 30 



1 



Marg I na I 
Tota I s 



Norn | na I 
Agreement 



Marg j na I 
Agreement 



Ligbtts 

Extems | on, 
of Kappa 



Self 
Other 
Nel ther 



1 
2 
3 



♦ 

♦ 
♦ 



22 
2 
2 



2 
30 
0 



0 
0 
2 



♦ 
+ 



2\ 
32 
4 



♦ 
♦ 

♦ 
♦ 



,917 

• 938 

• 500 



• 923 
1.000 

• 500 



,85| 
,866 
• ^83 



Marginal Totals 26 

♦ 



32 



♦ 
♦ 



60 



♦ 



Category Marginal Agreement! •808 
Cohen's Kappat »815 



* 408 

ERIC 



409 



/ 



Table IU 

Phase IV • IGE Evaluation Project 
Descriptive study - spring 1976 

Reliability on Pace 
Observer 40 Expert 80 
Scbool 801 Grade 2 
Week 9 Day 5 
WDRSD 



Category chosen 
by Expert 80 



Cat, 
ft 



Cateaory cbosen 
by Observer 40 



1 



Mara I na 
Tota I s 



Nomine I 
Agreement 



Marg | ne I 
Agreement 



Ugbt»s 
Extens 1 &n 

of Kappa 















♦ 




♦ 








Sel # 


1 


♦ 


24 


0 


0 


♦ 


24 


♦ 


1.000 


• 923 


1.000 






♦ 








♦ 




♦ 








Other 


2 


♦ 


0 


32 


0 




32 


♦ 


1*000 


1*000 


1.000 


















* 








Ne 1 tber 


3 




2 


0 


2 


♦ 


4 


♦ 


• 500 


• 500 


.483 














♦ 
























♦ 




♦ 








Marg I na 1 


Total s 


♦ 


26 


32 


2 




60 














♦ 








♦ 













410 



Category Marginal Agreementl .808 
Coben*s Kappat .938 



411 



9 

ERIC 



Table 215 



Phase IV - IOE Evaluation Project 
Descriptive Study - spring 1978 

Re M ab I II ty on Pace 
Observer 10 Expert 90 
School 801 Grade 2 
Week 9 Day 5 
DMP 



Category chosen 
by Expert 90 



se I f 
Other 
Ne I ther 



Cat, 
* 



Category chosen 
by Observer 10 



1 



Maro I ne 
Tota I s 



♦ 

1 ♦ 
♦ 

2 ♦ 
♦ 

3 ♦ 
♦ 



Maro I na I v Tota I s ♦ 

♦ 



33 
7 
1 

41 



1 

33 

2 

36 



1 
2 
4 



♦ 
♦ 
♦ 
♦ 
♦ 
♦ 

♦ 
♦ 



35 
42 
7 

84 



♦ 
♦ 
♦ 
♦ 
♦ 

♦ 
♦ 



Nominal 
Agreement 



,943 
• 786 
571 



Maro I na l 



Light »s 
Extens | on 



Agreement of Kappa 



,854 
• 857 
1*000 



• 888 
,625 
,532 



category Marginal Agreementl ,904 
Cohen's Keppal ,710 



412 



413 



\ ^ Table 216 

Phase**1v - lot tvaiuat Ion ^Project 
Descriptive Study - spring 1978 

Re I I ah I I I ty on Pace 
Observer 20 Expert 90 
School 801 Grade 2 
Week 9 Day 5 
OMP 



Category chosen 
by Expert 90 



sel f 
Other 
Nel ther 



Cat. 
H 



Category chosen 
by Observer 20 



1 



Mar g I na I 
Tota I • 



Nom | na I 
Agreement 





♦ 






♦ 






1 


♦ 31 


2 


2 




35 ♦ 


• 886 




♦ 






♦ 






2 


♦ 7 


33 


2 




42 ♦ 


• 786 




♦ 






♦ 






3 


♦ 2 


0 


5 


♦ 


7 ♦ 


• 714 




♦ 






♦ 








♦♦♦♦♦♦♦♦♦ 




















♦ 






s 


♦ AO 


35 


9 




84 ♦ 






♦ 













Marg I na I 
Agreement 



.^75 
,633 
.778 



Light «s 
Extens | on 
of Kappa 



,782 
,633 



68j> 



4H 



category Marginal Agreement! «829 
Cohen's Kappal ,69A 



415 



* Table 21? , 

Phase IV - IOE Evaluation Project 
Descriptive Study - spring 1978 

Re I | ah 1 1 I ty on Pace 
Observer 30 Expert Bo 
"school' 801 Grade 2 
Meek 9 Day 5 

OMP — J 



Category chosen 
by Expert 80 



Cat, 



Category chosen 
by Observer 30 



1 



Maro I na 
Total s 



Nom| na I 
Agreement 



Merg I na I 
Agreement 



UgM»s 
Extent | on 
of Kappa 



Sel f 
Other 
Ne I ther 



♦ 

♦ 

2 ♦ 

3 ♦ 
♦ 



Marginal Totals ♦ 

♦ 



47 
1 
0 

48 



12 
18 
0 

30 



3 
0 
3 



♦ 



62 



♦ 
♦ 

♦ 19 

4 \ , 

♦ 3 

♦ 



84 



♦ 
♦ 
♦ 
♦ 

♦ 



• 758 
,947 
1«000 



,774 

• 633 

• 500 



• 435 

• 918 
1*000 



category Marginal Agreement! ,636 
Cohen** Kappal ,6*5 



416 



417 



Table ^218 M 

Phase IV - IGE Evaluation Project ° 
Descriptive Study • Spring 1978 > 



Re I lab 1 1 I ty on Pace 
Observer 40 Expert So 
School 801 Grade 2 
Week 9 Day 5 
DMP 



Category chosen 
by Expert 80 



Cat. 
n 



Cateoory chosen 
by Observer 40 



1 



MarQlnal Nominal 
Totals Agreement 



♦♦♦♦♦♦♦♦ 



Marginal 
Agreement 



Set f 


1 


♦ 
♦ 


56 


6 


0 


♦ 62 ♦ 


.903 


.903 * 


.710 


Other 


2 


♦ 
♦ 


0 


19 


0 


♦ 19 ♦ 


1.000 


,760 


1 .000 


Nel ther 


3 


♦ 
♦ 


0 


0 


3 


♦ 3 ♦ 


1 .000 


1,000 


1.000 














♦♦♦♦♦♦♦♦♦♦♦ 








Marg 1 na 1 


Tota 1 s 


♦ 
* 


56 


25 


3 


♦ 84 ♦ 









Light*s 
Extens Ion 
of Kappa 



418 



Category Marginal Agreement! ,888 
Cohen*$ Kappal ,837 



4 m 



9 

ERJC 



» 



Table 219 

Phase Iv - IGE Evaluat ion Project 
Descriptive Study - Spring 1978 

Re I | ah | l | ty on croup I ng 
Observer 10. ExPert 90 
School 801 Grade 5 
Week 9 Day 4 
DMP 



Category chosen 
by Expert 90 



Category chosen by Observer 10 



Cat, 
* 



5 



Marg | na | 
Total s 



Norn I na I 
Agreement 



Marg | na » 



Light's 
Extens I on 



Agreement of Kappa 



Individual 
Pair 

Small Group (3»8) 

Large Group (>8» 

None of the above 

Marginal 



1 

2 
3 
4 
5 



♦ 
♦ 

♦ 
♦ 
♦ 



61 2 3 

8 7 2 

2 0 0 

0 0 0 

1 0 0 

Totals ♦ 72 9 
♦ 



0 
0 
0 
0 
0 



0 
0 
0 
0 

4 



♦ 
♦ 

♦ 

♦ 

♦ 
♦ 
♦ 

♦ 

♦ 
♦ 
♦ 



66 
17 
2 
0 
5 

90 



♦ 

♦ 
♦ 
♦ 
♦ 

♦ 
♦ 



• 924 

• 412 
0*000 

• 800 



.917 
.529 

• 400 

• 800 



,621 
,346 
,059 

.791 



420 



Category Marginal Agreement! ,662 
Cohens Kappai v 488 



/ 



421 



Table 220 

Phase IV • IGE Evaluation Project 
Descriptive Study - Spr Ino 1978 



•I 



Re l I ab I I I ty 
Observer 20 
School 801 
week 9 



on Grouping 
Expert 90 
Grade 5 
Day 4 
DMP 



-J 



Category chosen 
by Expert 9.0 



Ind I v I dua I 
Pal r 

Sma I | GrouP* (3-8) 

Large Group C^S) 

None of ^he above 



Category chosen by Observer 20 



Cat, 
u 



i v. 



% ♦ 



1 ♦ 

2 ♦ 

' * 

3 ♦ 
♦ 

4 ♦ 
♦ 

5 ♦ 
♦ 



Marginal Totals ♦ 

♦ 



59 
10 
1 
0 
0 



4 
7 
1 
0 

0 



Marginal Nominal 
Totals Agreement 



Lloht's 
Marginal Extension 
Agreement o* Kappa 









♦ 




♦ 






,523 


i 


0 


2 


♦ 


66 




,894 


• 943 


















,321 


0 


0 


0 


♦ 


17 


♦ 


,412 


♦ 706 












♦ 






• ,011 


0 


0 


0 




2 


♦ 


0*000 


• 500 








♦ 




♦ 








0 


0 


0 


♦ 


0 


♦ 














♦ 




♦ 






1 .000. 


0 


0 


5 


♦ 


5 


♦ 


1.000 


.714 








+ 




♦ 










422 



9 

ERIC 



category Marginal Agreementl ,716 
Cohen's Kappal .472 



123 



1 



Table 221 

Phase IV - IGE Evaluation Project 
Descriptive Study • Spring 1978 

Re i I ab I I I ty on Group I ng 
Observer ^0 Expert 80 
School 801 Grade 5 
Week 9 Day 4 

1 DMP 



Category thosen by Observer 10 



Category chosen 
by Expert 80 



Cat, 



1 



♦ 

Marginal Totals ♦ 

♦ 



32 



51 



Marginal Nominal 
Totals Agreement 



Ind I v 1 dua 1 




1 


♦ 
♦ 


26 


0 


0 


2 


0 


Pair 




2 


♦ 


2 


0 


0 


0 


0 


;roal | Group 


(3-8i 


3 


♦ 


0 


0 


0 


0 


0 


Large Group 


(>0) 




♦ 


2 


1 


0 


45 


0 


None of the 


above 


5 




2 


0 


0 


k 


6 



♦ 
♦ 

+ 



28 
2 
0 
48 
12 

90 



♦ 
♦ 

♦ 
♦ 



t 929 
0*000 

,938 
• 500 



Marg I na | 
Agreement 



• 875 

♦ 500 



Light's 
Extens I on 
o* Kappa 



t 889 

-•on 



• 941 

• 500 



9 856 
f 464 



category Marginal Agreement! t 704 
Cohen'i Keppal «750 



S3 



424 



ERIC 



425 



f 



Table 222 / 

Phase IV • IOE Evaluation Project 
Descriptive Study • Spring 1978 

Reliability on Grouping 
Observer 10 Expert 90 
School 601 Grade 2 
Week 9 Day 4 
WDRSD 



■vj 



Category chosen 
by Expert 90 



Cat , 
M 



Category chosen by Observer 10 

' s 1 2 

+ + + + + + + + + + 



Light's' 

Marginal Nominal Marginal Extension 
Totals Agreement Agreement ot Kappa 









♦ 
























Ind | v 1 dua | 




1 




5 


1 


0 


1 


0 


♦ 


7 




.714 


,857 


,686 








♦ 
























Pa 1 r 




2 


♦ 


0 


2 


0 


0 


-0 


♦ 


2 




1,000 


.667 


1,000 




















♦ 












Sma I | GrouP 


<3-8) 


3 


♦ 


0 


0 


0 


0 


0 




0 




























♦ 




♦ 








Large Group 


J>8) 


4 




0 


0 


0 


45 


0 


♦ 


45 


♦ 


1.000 


,918 


1,000 








♦ 












♦ 




♦ 








None of the 


abovt 


5 




1 


0 


0 


3 


8 




12 


♦ 


,667 


.667 


,621 








♦ 












♦ 




♦ 









Marginal Totals ♦ 



49 



66 



421". 



Category Marginal Agreementl ,777 
Cohen'S Koppal ,803 



427 



ERIC 



/ 



Table 223 

Phase IV - IGE Evaluation Project 
Descriptive Study - Spri.no 1978 



Re l fab i I I ty 
Observer 20 
School 801 
Week 9 



on Grouping 
Expert 90 
Grade 2 
Day 4 



WORSD 



Category chosen 
by Expert 90 



Cat, 



Category chosen by observer 20 



i 



Marg | na I 
Total s 



Noml nal 
Agreement 



Marg I na | 
Agreement 



Light's 
Extens I on 
ol Kappa 









♦ 












♦ 




♦ 








Xndl vl dual 




1 


♦ 


3 


2 


0 


1 


1 - 


♦ 


7 




• 429 


• 571 


,3*92 








♦ 












♦ 




♦ 








Pal r 

f. 




2 


♦ 


0 


2 


0 


0 


0 


♦ 


2 


♦ 


1*000 


• 500 


1,000 




















♦ 




♦ 








Sma I ; Group 


(3-8) 


3 




0 


0 


0 


0 


0 


4 


0 


♦ 














♦ 












♦ 




♦ 








Large Group 


<>8) 


4 




1 


0 


0 


44 


0 


♦ 


45 




,978 


• 938 


,919 




















♦ 




♦ 








None of the 


above 


5 




0 


0 


0 


3 


9 


♦ 


12 


♦ 


,750 


,833 


,705 




















♦ 




* 









Marginal Totals ♦ 



48 



10 



66 



♦ 



42b 



ERIC 



Category Marginal Agraementi ,711 
Cohan'i Kappa! #741 



ho 
-J 



429 



Table 224 

Phase Iv - IGE Evaluation Project 
Oescrlotlve Study - Spring 1978 

Re I | ab I I I ty on Group I ng 
Observer 40 Expert 80 
School 801 Grade 2 
Week 9 Oay 4 
WDRSD 



Category chosen 
by Expert 80 



Category chosen by Observer 40 



Cat, 

H 



Marg | na I 
Total s 



Nom I na I 
Agreement 



Marg I na | 
Agreement 



Light** 
Extens I on 
of Kappa 



























Ind I v 1 dua | 




1 




5 


0 


0 


0 


0 


♦ 


s 










♦ 
















♦ 


Pair 




2 




0 


0 


0 


0 


0 


♦ 


0 










♦ 


















Sma I | GrouP 


(3-8) 


3 


♦ 


4 


0 


33 


0 


0 




37 










4 












♦ 






Large Group 


t>8) 


4 




0 


0 


0 


0 


0 




0 


♦ 






♦ 


















None of the 


above 


5 


♦ 


0 


0 


o. 


0 


18 




16 


* 














1 . 

/ 










♦ 



Marginal Totals ♦ 

♦ 



33 



18 



♦ 
♦ 
♦ 



60 



♦ ♦♦♦♦ 
♦ 
♦ 
♦ 



1,000 



,89a 



1 .000 



,556 



,892 



1,000 



1,000 



\ 



\7/60 

s 

1,000 



Category Marginal Agreement) ,a!6 
Cohen»s Kappat ,881 



9 

ERIC 



Table 225 



Phase IV - IOE Evaluation Project 
Descriptive Study - Spring 1978 

Re | I obi I I ty on Grouping 
Observer \0 Expert 90 
School 801 Grade 2 
Week 9 Day 4 
OMP 



Category chosen 
by Expert 90 



Category chosen by Observer 40 



Cat, 



1 



Marg I na I 
Total s 



Nominal 
Agreement 



Marginal 
Agreement 



LlQht's 
Extern Ion 
of Kappa 



Individual 




1 


♦ 
♦ 


55 


1„ 


0 


0 


I 


♦ 


57 




• 965 


• 877 


,845 


Pair 




2 


♦ 
♦ 


6 


5 


0 


0 


0 




11 




• 455 


• 636 


,405 


Small GrouP 


(3-8) 


3 


♦ 


1 


1 


0 


0 


0 


♦ 


2 




o«ooo 


o f ooo 


0,000 


Large GrouP 


t>8) 


4 


♦ 


3 


0 


0 


4 


0 


+ 


7 




• 571 


• 571 


,550 


None of the 


above 


5 


* 

♦ 


0 


0 


0 


0 


7 


♦ 
♦ 


7 




1*000 


,875 


1,000 



Marginal Totals ♦ 65 

♦ 



a 



♦ 



84 



432 



Category Marginal Agreementl ,592 
Cohen»s Kappal ,658 



433 



Table 226 

Phase IV • IGE Evaluation project 
Descriptive Study - Spring 1978 



Re I I ab I I sty 
Observer 20 
School 801 
Week 9 



on Grouping 
Expert 90 
Grade 2 
Day 4 
DMP 



00 



Category chosen 
by Expert 90 



Category chosen by Observer 20 



Cat , 

n 



Margina I 
Totals 



Nom I na I 
Agreement 



Merg | na i 
Agreement 



Light's 
Extens I on 
of KaPPa 





















♦ 




♦ 








Individual 




1 


♦ 


52 




0 


0 


1 


♦ 


57 


♦ 


,912 


,934 


,680 








* 












♦ 




♦ 








Pal r 




2 


♦ 


3 


8 


0 


0 


0 


♦ 


11 




,727 


,917 


,682 




















♦ 












Sma 1 | Group 


{ 3-8) 


3 




2 


0 


0 


0 


0 




2 




0,000 


0,000 


0,000 




















♦ 




♦ 








Large Group 


(>8> 


4 


♦ 


k 


0 


0 


3 


0 


♦ 


7 


♦ 


,429 


,429 


,407 








* 












♦ 




* 








None of the 


above 


5 


♦ 


0 


0 


0 


0 


7 




7 


♦ 


1,000 


,875 


1,000 








♦ 












♦ 




♦ 









Marginal TotaH ♦ * 61 12 

♦ 



8 



♦ 
♦ 



84 



♦ 
♦ 



431 



Category Marginal Agreement! ,631 
Cohen»$ Kappa! ,651 



435 



ERIC 



i Table 227 



Phase IV - IOE Evaluation Project 
Descriptive Study -Spring 1978 



Reliability pn Grouping 
Observer ^0 Expert 80 
School 801 Grade 2 
week 9 Day 4 
DMP 



Category chosen 
by Expert 80 



Cat. 
U 



Category chosen by Observer 10 



Individual 


1 ♦ 


34 


7 


0 


1 


0 


Pair 


2 ♦ 


2 


0 

i 


0 


0 


0 


Smal) Group (3-8) 


3 ♦ 


8 


0 


4 


0 


0 


Cargo Group (>8) 


4 ♦ 


0 


0 


0 


18 


0 


None of the above 


5 ♦ 


1 


0 


0 


2 


1 


Marg | na 1 


Totals ♦ 


45 


7 


4 


21 


1 



Marg |na I 
Totals 



♦ 
♦ 

\ + 
+ 
+ 
♦ 
♦ 
♦ 

♦ 
♦ 



42 
2 

18 

4 



78 



Nominal 
Agreement 



• 810 
0«000 

• 333 
1.000 

• 250 



Marginal 
Agreement 



,933 

• 266 

• 333 

• 857 

• 250 



Light's 
Extens Ion 
of Kappa 



,550 
-,099 

*297 
1,000 
,240 



436 



Category Marginal Agreement* # 532 
Cohen * t Kappa t • 563 



ho 



437 



Table 228 



Phase IV - IGE Evaluation Project 
Descriptive Study • Sprint 1978 

Reliability on Grouping 
Observer 10 ExPert 90 
School 801 Grade 5 
Week 9 Day 5 
DMP 



ho 
a 
c 



Category chd*en 
by Expert 90 



indl vl dua 
Pal r 

Sma I | Group ( 3-8 ) 
Largo Group (>8) 
None of Oe above 



Cat , 
n 



Cateoory chosen by Observer 10 



1 



Mar 9 1 na I 
Tota I s 




Norn I na I 
Agreement 



.952 
.429 
.500 

.646 



LlQht»S 
Marginal Extension 
Agreement of Kappa 



,925 
,857 
,750 

,846 



,811 
,388 
,464 

,82S 



4 3 a 



Category Marginal Agreement! ,S'*5 
Cohen's Kappal ,685 



431) 



ERIC 



Table 229 



Phast IV «• IGE Evaluation Project 
Descriptive Study - Spring 1978 



Re I I ob I I 1 1 v on Group | ng 
Observer 20 Expert 90 
School 801 Grade 5 
Week 9 Day 5 
DMP 



Category chosen by Observer 20 



Category chosen 



Cat 



by Expert 90 


u 




1 


2 






♦ 






Ind | v 1 dua | 


1 


♦ 


55 








♦ 






Pair 


2 


♦ 




3 






♦ 






Smal | orouP (3<*8) 


3 


♦ 


2 


1 






♦ 






Large Group <>8> 


4 




0 


0 












None of the above 


5 


♦ 


0 


1 






















Mar g 1 na 1 


Tota 1 s 




61 


9 



2 
0 
5 
0 
0 



1 
0 
0 
0 
0 



0 
0 
0 
0 
15 

12 



Marg | na I 
Total s 



Nom I na I 
Agreement 

t 











* 




♦ 


f 887 












7 














♦ 


e 




• 6as 


♦ 










0 
















13 




• 923 


























♦ 









Marg l na | 



LlghMs 
Extens I on 



Agreement of Kappa 



.984 
,778 
• 875 
0,000 
,923 



,650 
,365 
,593 

,911 



category Marginal Agreement! ,712 
Cohen's Kappat ,666 

-ill 



Table 230 

Phase Iv - IOE Evaluation Project 
Descriptive Study - Spring 1978 

Re | I ab I I 1 1 y on Group | ng 
Observer 30 Expert 80 
School 801 Grade 5 
Week 9 Day 5 
DMP 



o. 
ho 



Category chosen 
by Expert 80 



Category chosen by Observer 30 



Cat* 
H 



Marg I na I 
Total s 



Nom I na I 
Agreement 



Marg | na | 
Agreement 



Light • s 
Extent I on 
of Kappa 



Ind I v | dua I 
Pair 

Sma I | Group <3~8) 
Large Group <*8) 
None of the above 



1 

2 
3 

5 



♦ 

♦ 
♦ 
♦ 
♦ 

♦ 



M'l 



Marginal Totals ♦ 

♦ 



22 
1 
0 
0 
0 

23 



4 
6 
2 
0 
1 

13 



0 
0 
0 
0 
0 



5 
1 
0 
<*7 
I 

54 



0 
0 
0 
0 
0 

















31 




• 710 


• 742 


• 610 


♦ 












♦ 


8 




♦ 750 


• 615 


• 706 
















2 




0.000 


0,000 


0.000 
















47 




i#ooo 


• 870 


1,000 


♦ 














2 




o«ooo 


OeOOO 


0,000 




























90 























Category Marginal Agreementi ,446 
Cohen's Kappaj ,715 



443 



ERIC 



Table 231 

Phase Iv - IGE Evaluation Project 
OescrlPtlve Study - Spring 1978 

Re I I ab i I I ty on Group | ng 
Observer 40 Expert 80 
School 801 Grade 5 
week 9 Day 5 
DMP 



Category chosen 
by Expert 80 



Category chosen by Observer 40 



Cat, 
n 



3 



Merg I na | 
Total s 



Norn I na I 
Agreement 



Marg | na | 
Agreement 



Ulght»s 
Extens I on 
of Kappa 



Ind I v I dua | 



Fair 



Sma I | Group (3-8) 



Large Group OS) 



hone of the above 



1 
2 
3 
4 
5 



+ 

♦ 

♦ 

♦ 
♦ 
♦ 



Marginal Totals ♦ 

♦ 



28 
5 
2 
5 
0 

40 



0 
2 
0 
0 
0 



0 
0 
0 
0 
0 



3 
1 
0 
42 
0 

46 



0 
0 

b 
o 

2 



♦ 

♦ 

♦ 

♦ 
♦ 
♦ 

♦ 
♦ 
♦ 

♦ 

♦ 



31 
8 
2 

47 
2 

90 



♦ 
♦ 
♦ 
♦ 
♦ 
♦ 
♦ 
♦ 
♦ 

♦ 
♦ 
♦ 



.903 
.250 

0,000 
,894 

1,000 



.775 
• 250 

0,000 
,979 

1,000 



,826 
,233 

0,000 
,782 

1,000 



441 



Category Margin?! Agreement! ,601 
Cohen»s Kappa i ,692 



Table 232 

Phase IV - IGE Evaluation Project 
Descriptive Study - Spring 1978 

Re M ab I I I ty on Group I rig 
Observer 10 Expert 90 
School 601 Grade 2 
Week 9 Day 5 
WORSO 



00 



Category chosen by Observer 10 



Category chosen 


Cat • 








by Expert 90 


* 




1 


2 
















♦ 






Individual 


1 


♦ 
♦ 


16 


0 


Pair 


2 




2 


7 












Smal | Group (3*»8) 


3 


♦ 


1 


0 


Large Group (>8) 


4 




5 


2 






♦ 






None of the above 


5 


♦ 


0 


0 


Marg 1 na 1 


Total s 


♦ 


24 


9 



0 
2 
0 
0 
0 



i 

0 
0 
21 

0 

22 



1 
0 
0 
0 
8 



Marg | na I 
Total* 



♦ 
♦ 



18 
11 
1 

28 
8 

66 



♦ 
♦ 

♦ 

♦ 



Norn I na I 
Agreement 



# 889 

• 636 
0*000 

• 750 
1*000 



44K 



Category Marginal Agreement! ,749 
Cohon»s Kappat f 705 



Marg | na | 
Agreement 



.750 
• 818 
,500 
,786 
,889 



Ug*t»s 

Extens Ion 
of Kappa 



,825 
,579 

.,031 
,625 
1,000 



447 



9 

ERIC 



Table 233 

Phase IV - -IGE Evaluation project 
Descriptive Study - Spring 1978 

Reliability on Group I no 
Observer 20 Expert 90 
School B01 Grade 2 
Week 9 Day 5 
WORSD 



Category chosen 
by Expert 90 



Category chosen by Observer 20 



Cat, 
H 



Marg | na I 
Total s 



Norn I na I 
Agreement 



Marg | na I 
Agreement 



Light's 
Extens I on 
of Kappa 









♦ 














♦ 








Indl v |dua| 




1 


♦ 


13 


1 


2 


2 


0 i 


y 18 




• 722 


,857 


,593 








♦ 






















Pair 




2 


► 


2 


9 


0 


0 


0 i 


► 11 


♦ 


,818 


,646 


,774 


Sma | | Group 


<3-B) 


3 


♦ 
♦ 


0 


0 


1 


0 


0 i 


► 1 


♦ 
♦ 


1*000 


,250 


1,000 


Large Group 


(>8> 


4 


♦ 
♦ 


6 


3 


1 


16 


0 i 


► 28 




«643 


,714 


,488 


None of the 


above 


5 


♦ 
♦ 


0 


0 


0 


0 


8 4 


► 8 


♦ 
♦ 


1*000 


1*000 


1,000 








♦ 























21 



13 



SO 



6 



Marginal Totali ♦ 



♦♦♦♦♦♦♦♦♦ 
♦ 

66 ♦ 

♦♦♦♦♦♦♦♦♦ 



448 



Category Marginal Agreement I $734 
Cohan 1 1 Kappa I 1 650 



OO 



4 1 9 



9 

ERIC 



Table 234 N 

Phase Iv - IOE Eva I uet I on* Project 
Descriptive study - Spring 1978 K 

CC 

cr 

Re I I ab I I I ty on r,rouP I ng 
Observer 30 Expert 80 
School 801 Grade 2 
Week 9 Day 5 
WDRSD 



Category chosen by observer 30 



Category chosen Cat. 
by Expert 80 » 



Individual 1 

Pa|r 2 

Sma I | orouP ( 3-8) 3 

large GrouP <>8) A 

None of the above S 



♦ 

♦ 

*■ 

♦ 



Marg I n« | T< ta I s ♦ 



21 
0 
0 

2 
2 

25 



0 
3 

0 
0 
0 



0 
0 
0 
0 

0 



0 
0 
0 
30 
0 

30 



Marg I na 



Now I na I 



Marginal 



+ 
♦ 
♦ 



60 



♦ 
♦ 
♦ 



U I ght • s 
Extens I on 



5 




Total s 




Agreement 


Agreement 


of Kappa 








♦ 








0 




21 


♦ 


1*000 


.840 


1.000 








♦ 








0 




3 




It ooo 


1.000 


1 ,000 
















0 




0 












♦ 












0 




32 


+ 


.938 


.938 


,875 
















2 




4 




• 500 


♦500 


,483 

















450 



451 



Category Marginal Agreement! ,Rl9 



Cohen's Kappai .886 



Table 2Y-> 

Phase IV - 101 Evaluation Project 
Descriptive Study - Spring 1978 

Pel i ah I I I t v on Group i no 
Observer 40 FxPert 60 
School 801 Grade 2 
week 9 Day 5 
WDRSD 







Category 


chosen 


by 


Observer 


40 










L 1 ah t ' 8 


Category chosen 


Cat 














Maro I na 


1 


Nom 1 na 1 


Marg j na i 


Extens 1 on 


by Expert 80 


* 




1 


2 


3 


4 


5 




Tota 1 i 




Agreement 


Agreement 


of Kappa 


















♦♦♦♦ ♦ 




























♦ 








Ind ! v 1 dua 1 






21 


0 


0 


0 


0 


♦ 


21 


♦ 


1.000 


,913 


1 ,000 


















♦ 




♦ 








Pa I r 






0 


3 


(■ 


0 


0 


♦ 
♦ 


3 




l.ooo 


1 .000 


1,000' 


Sma S | Group ( 1- M * 


3 




0 


0 


0 


0 


0 




0 


< 






-- — •» 


















♦ 




♦ 








large Group OHj 






0 


0 


0 


32 


0 




32 


♦ 


1.000 


1.000 


1.000 


None of the* above 




♦ 




0 


0 


0 


2 


♦ 

+ 
* 


4 


♦ 


.SCO 


,b00 


,463 






♦ 

+ 4 ♦ 


















1 


0 


3? 


2 




6>0 


♦ 
























+ 




♦ 



























?;at.f"QO?v Margj ! n^| Agrettrotftt* I #Mm.a 
f,' S ^Hn * i Kappa I * 



er|c 



Table 21b 

Phase IV • IGF Evaluation Project 
Descriptive Study - Spring 1978 

Reliability on Grouping 
Observer 10 Export 90 
School 801 Grade 2 
week 9 Day 5 
DMP 



a 

or 



Category chosen 
by Expert 90 



Individual 
Pal r 

Sma I | Group ( 3-f ) 
Large GrouP (>e> 
None of the above 



Cat . 
U 



Category chosen by Observer 10 
1 

♦ 

♦ 3 ) 1 
♦ 

♦ 05 
♦ 

♦ 00 

♦ 56 
+ 

♦ 1 0 



Light's 



3 4 






Marg | na I 


Norn 1 na I 


Mara i na | 


Extent 1 on 


5 




Tota I » 




Agreement 


Agreement 


of Kappa 


























0 1 


1 


♦ 


33 




• 909 


• 917 


• 641 






♦ 












0 0 


0 




5 


♦ 


i t000 


• 417 


1.000 










4 








0 0 


0 




0 






0 f ooo 




















C 26 


2 




39 




• 667 


• 692 


t 509 






+ 












2 0 


4 




7 




• 571 


1*000 


,532 






♦ 












♦♦♦♦♦♦♦ 
























2 27 


7 




84 


< 
















♦ 









4 r> t 



Category Marginal Aoreementj ,6n^ 
Cohan's Kappa I ,661 



45;) 



0 

ERIC 



Table 237 

Phase Iv - IGF Evaluation Project 
Descriptive Study - Spr I no 1978 

pel I a b I I | t y on Group I ng 
Observer 20 Expert 90 
School 801 Grade 2 
week 9 Day 5 
DMP 



Category chosen by Observer 20 



Category chosen Cat, 
by Expert 90 n 



1 



Marg | na i 
Total s 



Norn I na I 
Agreement 



Marg l na | 
Agreement 



Light 
Extens I 5r> 
of KapPt 



Individual 1 

Pair 2 

Sma I | Group (3-8) 3 

Large Group (>8) 4 

None of the above 5 



+ 
♦ 
♦ 
♦ 
♦ 

♦ 
♦ 
♦ 

* 
♦ 



Marginal Totals ♦ 

♦ 



28 
0 
0 
4 



I 

4 
0 

1 

0 



0 

1 

0 
0 
2 



2 
0 
0 
32 

0 



32 6 3 34 



2 
0 
0 
2 
5 



♦ 
♦ 

♦ 

♦ 
♦ 
♦ 
♦ 

+ 
♦ 

♦ 

♦ 



33 
5 
0 

39 
7 

84 



♦ 
♦ 

♦ 

♦ 
♦ 
♦ 

♦ 
♦ 
♦ 



.848 
.800 

,821 
.714 



,970 
,833 
0,000 
,672 
,778 



,755 
,765 
••»• 
,698 
,680 



++++++++ 



9 

ERIC 



category Marginal Agreement! ,691 
Cohen's Kappai ,725 



•V.)t" 



45? 



VX; 



Table 238 



Phase IV • IGE Evaluation Project 
Descriptive Study - Spring 1978 ^ 

o 

ue | | ab I I l ty on Group | ng 
Observer 30 Expert 80 
School 001 Grade 2 
Week 9 Day 5 
DMP 



Category chosen by Observer 30 



Category chosen Cat, 
by Expert 80 * 



Individual 1 

Pair ? 

Smo I | GrcuP (3-8) 3 

Large GrcuP (>8) 4 

Nona of the above 5 



♦ 

4 

4 
4 
4 

4 



Marginal Totals 4 

4 



41 

3 
0 
0 
0 

44 



i 
9 

1 

0 



1 1 



2 
0 
1 
0 
0 



4 
0 
1 

18 

0 

3 23 



Marg | nai 
5 Totals 

44444444444444 



Nominal Marginal 
Agreement Agreement 



0 
0 
0 
0 
3 



4 
4 
4 
4 
4 
4 
4 
4 
4 
4 
4 

4 
4 
4 



48 
12 

3 
18 

3 

84 



4 
4 
4 
4 
4 
4 
4 
4 
4 
4 
4 

4 
4 

4 



+444444 4 444444*444444444444444444444444 

Crtegory Marginal Agreementl ,923 
Cohan's Kappal #770 



,750 
,333 
1,000 
1 ,000 



,917 
• 917 
1,000 

,703 
1,000 



Light »s 
Extent 1 on 
of Kappa 



, 694 
,712 
,309 
1,000 
1,000 



453 



Table 239 

Phase iv • it»t tvaiuation Project 
Descriptive Study - Spring 1978 

Re I I ob I I I ty on Grouping 
Observer 40 Expert 80 
School 801 Grade 2 
week 9 Day 5 
DMP 



Category chosen by Observer 40 



Category chosen 
by Expert 80 



Ind I v I due I 
Pa I r 

Small Group (3-0) 
Large Group (>ej 
None of the above 



Cat. 
n 



♦ 



1 ♦ 
♦ 

2 ♦ 
♦ 

3 ♦ 
♦ 

4 ♦ 

5 ♦ 
♦ 



Marg I na I Tota I s ♦ 

♦ 



43 
5 
0 
0 
0 

48 



0 
7 
2 
0 
0 



3 
0 
0 
0 
0 



2 

0 

1 

18 

0 

21 



0 
0 
0 
0 
3 



Marg I na I 
Total s 



Light's 

Nominal Marginal Extension 
Agreement Agreement of Kappa 



♦ 
♦ 
♦ 

♦ 

♦ 
♦ 

♦ 

♦ 

♦ 
♦ 



48 
12 

3 
16 

3 

84 



♦ 
♦ 
♦ 

♦ 

♦ 

♦ 
♦ 
♦ 



• 896 
,583 
0*000 

i;ooo 

1»000 



1.000 
.750 

1.000 
,857 

1.000 



,757 
,533 
-.,037 
1,000 
1,000 



Category Marginal Agroementl ,921 
Cohen's Kappal ,743 



K3 



4K0 



461 



Table 240 



Phase IV - I0E Evaluation Project 
Descriptive Study - spring 1978 

Re I I ab | I 1 1 y on Mater I a 1 1 
Observer 10 Expert 90 
School 801 Grade 5 
Week 9 Day <♦ 
DMP 



Mater tall PaPer/penc I I 



Category chosen 
by Expert 90 



Cat . 
* 



Cateoory chosen 
by Observer 10 



1 



Merg I na 
Tota I s 



Nom | na I 
Agreement 



Merg | na I 
Agreement 



Light it 
Extens | on 
of Kappa 



























Used 


1 


♦ 


61 


21 


0 




62 


♦ 


• 74* 


• 756 


• 177 






♦ 




















Not used 


2 




1 


2 


0 




3 




• 667 


• 125 


• 545 


























Other 


3 




0 


1 


4 




5 




• 800 


• 600 


• 791 








































































Marg I na 1 


Total S 




62 


2* 


<♦ 




90 



































.+++++++++ 



463 



category Marginal Agraementi t 560 
Cohen't Kappal #29? 



ERIC 



Table 241 

Phase IV - IGE Evaluation Project 
Descriptive Study - Spring 1978 

Re I I ab I'M ty on Mater I a I s 
Observer 20 Expert 90 
School 801 Grade 5 
week 9 Day <» 
DMP 



Mater I a I I PaPer/penci I 



Ceteqory chosen 
by Expert 90 



used 

Not used 
Other 



Cat • 



1 

2 
3 



Marginal Totals 



Cateaory chosen 
by Observer 20 



80 
2 
0 

62 



1 1 
0 1 
0 5 



Marginal Nominal Marginal 
Totals Agreement Agreement 



82 
3 
5 

♦ ♦■♦■♦•♦>♦♦♦ 
90 



.976 
0.000 
1.000 



1 .000 
, .333 
.71^ 



Llghttt 
Extens | on 
of Kappa 



,726 
-.011 
1.000 



9 

ERIC 



46'. 



category Marginal Agreementl .693 
Cohen's Kappa! .66* 



465 



Phase IV - I GE Evaluation Project 
Oescriptlve Study - Spring 1978 

Re I I ab M I ty on Mater I a I ft 
Observer //o Expert 80 
School 801 Grade 5 
Week 9 Day <f» 
DMP 



Material! Paper/pencil 



Category chosen 
by Expert 60 



Category chosen 
by Observer lo 



Cat . 



Mara I na I 
Total s 



Nom | na I 
Agreement 



Marg | na I 
Agreement 



L igbt • s 
Extens Ion 
>f Kappa 



♦ ♦ ♦ 

Used 1 t 12 0 ♦ 60 ♦ 

♦ ♦ ♦ 
Not used ? * 2 16 0 ♦ lfi ♦ 

Othpr 3 ♦ 1 5 6 ♦ 12 ♦ 

+ + ♦ 

Marginal Totals ♦ 51 3? 6 ♦ 90 4 



• 800 
,889 

,500 



,850 
,500 



,538 
,<»6<< 



Category Marginal Agreement? ,632 



Coder's Kappal ,5flfl 



Table 243 



Phase IV - I GE Evaluation Project 
Descriptive Study - spring 1978 

Re I l ob I I 1 1 y on Mater I a I s 
Observer 10 Expert 9o 
School 801 Grade 2 
Week 9 Day U 
WDRSD 



Material! Paoer/pencll 



Category chosen 
by Expert 90 



Used 

Not used 
Other 

Mara l na 



Cat, 
H 



Cateaory chosen 
by Observer 10 



1 



Marg I na 
Total s 



♦ ♦ ♦ 

1 ♦ 9 1 0 ♦ 10 ♦ 

♦ ♦ ♦ 

2 ♦ 2 42 0 ♦ <»<♦ ♦ 

♦ ♦ ♦ 

3 ♦ 1 3 8 ♦ 12 ♦ 

♦ ♦ ♦ 

♦ ♦ ♦ 

Totfc . ■ * 12 8 ♦ 66 * 

4. 4. 4 



Nom | na I 
Agreement 



• 900 
,955 
,667 



Marg I na I 
Agreement 



• 833 



.95 



,667 



Light i s 
Extens I on 
of Kappa 



,878 



,850 



,621 



category Marginal Agreement! 

Cohan's Kappal ,78? 



*19 



Table 2 4 4 

Phase IV - I GE Evaluation Project 
Descriptive Study - SPr inQ 1978 

Re I |ab I M ty on Mater I a l s 
Observer 20 Expert 90 
School 801 Grade 2 
week 9 Day 4 
WDRSD 



Material I Paper/pencil 



Category chosen 
by Expert 90 



Used 

Not used 

Otner 



Category chosen 
by Observer 20 



Cat . 
ft 



4 

1 ♦ 

4 

? ♦ 
* 

3 4 



Mar Q i I Nom | nfl I 
Totals Agreement 



Marginal Totals + 



+ ♦ 

1 1 ♦ io ♦ 

* ♦ 

0 * ♦ 

♦ ♦ 

3 9 ♦ 12 ♦ 

4 
4 

A 10 ♦ 6f> 4 

4 4 



6 

0 
C 



iflOO 
1 .000 
.750 



L ig^t • s 
Marginal Extension 
Agreement of Kapoa 



.800 
.917 
.833 



.772 

1 .000 
,705 



++++4 44444*444*44 4 44444444444 



category Marginal Agreement I • 



850 



roar's Hflpp* l ,8.1^ 



Table 245 

Phase IV - IQE Evaluation Project 
Descriptive Study - SPrlnQ 19T8 

Re I lab I I I tv on Mater I a I 8 
Observer 40 Expert 60 
School 801 Grade 2 
Week 9 Day 4 
WDRSD 



Material! Paper/pencil 



Category chosen 
by Expert 80 



Used 

Not used 
othar 

Mar o I n« 



Cat o 
n 



Cateaory chosen 
by Observer 40 



1 



M a r g I na 
Tota I s 



Nom | na I 
Agreement 





♦♦♦♦♦♦♦♦♦♦♦♦4 


























♦ 




#000 


1 


♦ 25 


0 


0 




25 




1 
















.471 


2 


♦ 9 


8 


0 




17 








♦ 






■f 










3 


♦ 0 


0 


IB 




18 




1 


• 000 














♦ 







♦ ♦ ♦ 

Totals ♦ 34 8 13 ♦ 60 ♦ 



L I ght » 8 
Marginal Extension 
Agreement of Kappa 



,735 1.000 
,471 .389 
1,000 1.000 



9 

ERIC 



•t t 



category Marginal Agreement I .735 
Cohen's Kappa! .764 



<17:< 



Table 246 



Phase IV - K,E Evaluation Project 
DescriDtlve Study - SPMnQ 1978 

Re I I ab I I I ty on Mater I a I $ * 
Observer IQ Expert 90 
School 801 Grade 2 
Week 9 Day 4 
DMP 



Material! PaPer/pencll 



Category chosen 
by Expert 90 



Cat , 



Cateaory cKOsen 
by Observer 40 



1 



Mar 0 I na 
Tota I s 



Nom I na I 
Agreement 



Mar g I na I 
Agreement 



Light is 
Extens I on 
o* Kappa 



Used 1 ♦ 76 0 1 + 77 ♦ 

Not used ?♦ 0 0 0* 0* 

Other 3 ♦ 0 0 7* 7* 

♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 
♦ ♦ ♦ 

Maralrrtl Totals ♦ 76 0 8 ♦ 84 ♦ 



,987 



1 ,000 



987 



.875 



• 864 



1 .000 



475 



cateaory Marginal Agreement! 



931 



Cohen's Kappa! 



.927 



Table 247 



phase IV - IGE Evaluation Project 
Descriptive Study - SPr ln« 1976 

Re I I sb I I ' tv on Materials 
J Observer 20 Expert 90 

School 801 Grade 2 
week 9 Day <* 
DMP 



Material l PaPer/pencll 



Category chosen 
by Expert 90 



used 

Not used 

Cthar 



Cateaory chosen 
by Observer 20 



Marginal Tots 



M a r o I n a 
Total s 




Nominal Marginal 
Agree'Ksnt Agreement 



♦ 967 



1*000 



.987 



.875 



L I flht * S 
Extens | on 
of Kappa 



• 864 



1.000 



0«teaery na r B'. n*M Agreement ! 



.931 



Table 248 



J 



Phase IV • X GE Evaluation Project 
Descriptive Study - Sprlno 1978 

Re l labl I 1 1 v on Materials 

Observer 4(5 Expert 80 

School 80 I s ) Grade 2 

Week 9 Day 4 

0 ' OMP v 



v. 



c 
c 



Mater | a f 3 Paper/pencl I 



Category chosen 
by Expert 80 



Used 

Not used 
Other 



in 



Category chosen 
by Observer 10 



MarG I na I Nominal 
Totals Agreement 



LlQht«i 
Marc I na I . Extension 
Agreement of Kappa 



+ + + 4. + + + + . >>** + «>'* + 4"»"*''»'f* + ♦♦♦♦♦♦ 

1 + US 1 0 '4? ♦ 
♦ ♦ * 

2 ♦ 1 26 0 ♦ 2T ♦ 



Marginal Totals ♦ 



3 + 1 2 1 ♦ * ♦ 

+ ♦ ♦ 

. . 4, 4, + + + 4. + + + + + + + * + + + + + + + + + * + + 

♦ * * 

48 29 1 ♦ 78 ♦ 



♦ ♦ + + 4 



,979 
,963 
.250 



Category Marginal Agreement 1 ,720 
Cohen's Kapoal ,872 



,979 
• 931 
,250 



• 9*5 
,941 

,240 



•17 J 



ERIC 



Table 249 

t>hess !v - IGE Evaluation Project 
pescrlptlvfl Study - Spring 1978 

pel lability on Material! 
Observer 10 -Expert 90 
. School 801 Grade 5 
, Week 9 Day 5 a 
/ *©MP 



"v. 



Mater I a I t Paper/pencl I 



IV 



Category chosen 
by. Expert 90 



Used 

Not used 



Other 



Category chosen 
by Obsenver 10 



Cat. 
0 



1 ♦ 
♦ 

2 4 

4 

3 ♦ 



72 
0 
2 



Marginal Totals ♦ 

■ ♦ 



5 

0 



0 
0 
11 



Marginal Nominal Marginal 
Totals .Agreement Agreement 



4 
♦ 

4 
♦ 

4 

♦ . 



77 
0 
13 



4 
♦ 
♦ 

4 

4' 




,93$ 



• 846 



,961 
Q.,000 
, 846 



Ughtts 
Extens ion 
Of kappa 



,635 



025 



4b0 



Category Marginal "Agreement l ,602 
Cohan ' s Kapvpa I ,721 



' 481 



Table 250 

Phase TV - IGE Evaluation Project 
Descriptive Study - SPMnfl 1978 

Re I I eb Ml I ty on , Mater I a I s 
Observer 20 * Expert 90 
Sohoo I 801 ' Grade 5 [ 
Week 9 Day 5 , 
„. DMP 



Mater lYl I Paper/penc II' 



C 1 
ho 



Category chosen 
by Expert 90 



Cat. 



Category chosen 
by Observer 20 



Marginal Nomina 



Mara I na I 



Llght»s 
extension * 



Totals Agreement Agreement of Kbppa 



Used 

Not used 
Other - 



Marg I na I 



♦ 

1 ♦ 

■ ♦■ 

2 ♦ 

3 ♦ 
♦ 



Totals ♦ 



7* 
o 

l 

77 



1 

0 



0 
0 
1-2 

12 



♦ 
♦ 

+ 
♦ 



77 
0 
13 

90 



♦ 
+ 
♦ 
♦ 
♦ 



'•987 



*923 * 



1 e 0 0.0 

0.000 
.923 



.910 



.9-11 



483 



category Marsinal Agreementl »6<»1 
Cohen's Kappal .911 



*■ Table 251 

A 

Phase IV - iGE Evaluation Project 
Descriptive Study • Spring 1978 

Reliability on^ Mater I a I s 
Observer 30 ' Expert 80 
School 801 . -Grade S 
J- Week 9 Day 5 

' DMP \ 



Category chosen 
by'ExPert 80 



Material! Paper/pencll 

♦ * 



Cat. 
u 



Category chosen 
by. Observer 30 



1 



Marfl I na I 
, Tote I s 



Nominal Marginal 
Agreement Agreement 



U I p'Kt ♦ s 
Extens | on 
of Kappa 















♦ 




♦ 




•Used % 


1 


+ 


75 


1 


0 




76 


♦ 


%987 






♦ 








4* 




♦ 




Not used 


-2 




10 


2 


0 


4 


12 


♦ 


• 167 






♦ 








4 








Other 


3 


♦ 


1 


1 




♦ ' 


2 


♦ 


0t000 


• 
















♦ 





Marginal Totals «► 

' ♦ 



86 



♦ 



90 



♦ 
♦ 
♦ 



'.884 




,704 
• 126 
0.00J) 



Category Marginal Agreement! .406 
Cohan's kappal .228 



4a 4 



485 



Table 252 

Phase IV - IGE . Evaluation Project 
D*.sxr lot I ve Study - spring 1978 

Reliability on Materials 
Observer 40 Expert Bo 
School 801 Grade S 
Week 9- Day 5 • 
DMP 



Mater | at t PaPer/penc 



o 



Category chosen 
by Expert 80 



Category chosen 
by Observer -40 




Light's „ 
Extens Ion 



Marginal Nominal Marginal 
^ofcels Agreement Agreement of Kappa 



,934 
1.000 
1.000 



• 93 V 
.706 

1. 000 



,686 



i.ooa 



1,000 



L 



487 



category Marginal Agreement! ,880 



Cohen's Kappal ,820 



/ v. 



Tabl.e 253 

Phase IV - IGE Evaluation 'Project 
Descriptive Study - SPrlnfl 1978. 

pel | I I I ty on Materiel s 
Observer 10 Expert 90 
School 801 Grade 2 
Week 9 /Day 5 
WDRSP 



Category chosen 
' by Expert 90 



Used. 
Not uHed 
Other 



Cat. 



Marginal Total* ♦ 

V ♦ 



Mater (alt Paper/penc I ! 



Cateaory chosen 
by Observer 10 



Marginal Nominal Marginal 



Uloht»s 
EXtens I 6r 



2 3 Totals Agreement Agreement of Kappa 

. .585 »585 .3*8 

1,000 .515 / 1.000 

1.000 - .889 1,000 




\ 



488 



category Marginal Agreementi 
Cohen* » Kappat .590 



1 66 3 



489 



is 



T. 

\ 



\ 



7T 



♦ 'A 



. • v - 

> 

S r Table 254 

Phase IV - IQE frya.luatlon Project 
( Descriptive Study - Spring 1978 



'V* 



hi 



c 



r 



Re I I ab I I I ty on Mater l a I s 
'Observer 20 Expert .90 
School 801 ' Grade 2 
Week 9- ■ Day /6 
WDRSD ' 



water | a| | .Paper/pencl I 



' Category chosen 
by Expert 90 



Cate<*ory* chosen 
-by Observer 20 



r 



Cat « 
9 



\ ' 



« 



Mara I na I 
Tota I s 



Nom | na 



Marg I na I 



Llghtt's 
EXten's ion' 



Agreement Agreement of Kappa' 



c » 

Used 


« 

1 + 

• 


• 37 


4 , 


0 


♦ 


41 ♦ 


.902 


,.976 


.752 


> 


* 1 f 


tsLo t used 


2 ♦ 


1 

* 3 


14 


0 




17 ♦ 


.824 


> 

.944 ... 


,757 






Other \ 


3 ♦ 


0 


0 


6 


♦ 


8 • V 


l/boo 


1.000 


1,000 








+++++++++ 






4- 












# 


Marg I na 1 


Totals ♦ 


AO 


18 


8 n 




66 ♦ 










^ 


• 


T 
















* 






% 






f 






* 


* 




431 





na/f 



category Marolna/I Agreement} ,973 
Cohen's Kappal .803 



ER?C 



4 r 

Table 2$ 5 

Phase IV - IOE Evaluation Prdject 
• Descriptive Study - 'spr|r»fl 1978 

Re I I ab 1 1 I ty on Materials 
Observer 30 Expert 80 
School 801 :. Grade 2* 
week 9 " Day 5 
WDRSD 



Mater lalT PaPerVpenc I I 



Category chosen 
by Expert 80 



Used 

Not used 
Other 



Cat, 



1 
2 
3 



Marginal Totals 



Category Chosen 
by Observer 30 



1 

51 

< 0 
2 



l 

53 



6 



j Lights 

Marolnal Nominal Marginal Extension 
TotaH Agreement Agreement of Kappa 



si 



♦ 
♦ 
♦ 



It 030 



5 <», ♦ v * 1.000 

4 . ♦ tS00 
♦ 

♦ 

•o 60 - ♦ 
♦ 

♦♦♦♦♦♦♦♦♦ 



,962 

1,0 0^0 
• 500 



1.000 
1,000 
,483 



Category Marginal Agreementl ,821 
Cohan»s Kappol ,861 



492 



493" 



9, Table- 256' 

phase IV- I<*E Evaluation Project 
Descriptive Study Spring 1978 

Pel lab! Tltv on Materials 
* , Observer 40 Expert 80 
School 801 ' Grade 2 
Week 9 Day 5 
WDRSD 



«». 



> Mater I a I t Paper/pencl I 



Category chosen 
by Expert 80 



4<J4 



used 

Not used 

*? ' « 
Other 



£at« 



Category chosen 
by Observer 40 



1 



3 



Marginal Nominal 
Totals Agreement 



1 ♦ 

2 ♦ 
♦ 

3 ♦ 

+++++++++++++ 

Marginal. Totals ♦ j 54 4 

♦ 



SI 


0 ' 


0 




v 51 


1 * 


4 


.or 


♦ 


5 


2 


• 0 


2' 


♦ 


* 4 



♦ 
♦ 




Category Marginal Agreementi 
Cohen's Kappat ,TBo 



1.000 

• 800 

• 500 



Marginal 
Agreement 



• 944 

• 800 

• 500 



,748 



l|ght»S~ 
Extens l on 
of Kappa 



1,000 
,786 
',483 



495 



\ 



Table 257 

Phase IV • idr Evaluation Project 
Detcrl.pt I ve, Study - Spring 1978 

j Rel |ab| I 1 1 v on Materials 
i Observer 10 Expert 90 
School 801 Grade 2 
Week 9 Day 5 
DMP' 



Category chosej 
by Expert 90 



used 

Not used 
Other 



Cat • 

n 



Material t Paper/pencl I 

> 



Ciategory chosen 
by Observer 10 



1 ♦ 
♦ 



3 ♦ 
♦ 



LlOht'S , 

Marginal Nominal Marginal E^nti on- 
Total « Agreement Agreement^ o* Kappa 



AO 


5 


2 i 


► 47 


1 


28 


1 < 


30 


4 


2 


4 


► 7 



++++++ ♦♦♦♦♦♦♦ ♦ ♦<►♦♦♦ 

♦ <: 
Marginal TotaM ♦ 42 35 7 



♦ 
♦ 

♦ 

♦ 

84 * 

♦♦♦♦♦♦♦♦♦ 



• 851 

• 933 

• 571 



• 894 

• 857 
1*000 



\ 



,702 
• 886 
,532 



category Marginal Agreement! ^IT 
Cohen** Kappal »747 



496 



o 



497 



''Table 258 

Phase IV - K'E Evaluation Project 
Descriptive Study - spring 1.978 

Re I lab 1 1 ItV on Mater lal s 
Observer 20 Expert 90 
School 801 Grade 2 
Week 9 Day 5 
DMP * 

Material I Paper/pencil 



c 



Category chosen 
by Expert 90 



used 

Not used 

Other 



Cat * 
n 



Category choseo 
by Observer 20 



1 



49B . 



♦ 

♦♦♦♦♦ 
♦ 

Marginal Totals ♦ SO 

♦ 

+++++++ 



2 i 



Marg I na I 
Total s 




Nominal * 



Marginal 



Llght»s 
Extens ion 



Agreement Agreement of Kappa 



r 


♦ - 


43 


1 


3 




♦ 








2 


♦ 


6 


23 


1 




♦ \ 








3 




1 


1 


S 



35 



4 
♦ 

♦ 
♦ 
♦ 

♦ 
♦ 



*7 

H 

7 

.84 



♦ 
♦ 
♦ 

♦ 
♦ 
♦ 

♦ 



♦ 915 

• 767 



71<» 



• 940 

,83~2f 
.778 



,790 

4 

,668 
,680 



499 



category Marginal Agreementl 
Coheres Kappal ,719 



.850 



Table 259 

pha-se IV*- lot Evaluation Project 
Descriptive Study - Spring 1978 

Rehlab! I I ty on Mater lals 
ObWrver 30 Expert 
S*cho6l 801 Orade 2* 
week 9 Day 5 
DMP - 



Mater let i Paper /pencl I 



Cateaory chosen 
by Observer 3Q- 



Category chosen 


Cat. 




1 




by Expert 80 

* 


# 




2 


v 




+ 






Used 


1 


♦ 


63 


3 


% Not' used , 


2 




0 


15 


Other 


3 


♦ 


to 


0 



Marginal Nominal* Mara; i rial 
Totals Agreement Agreement 



0 
0 
3 



♦ 

Marginal Totan ♦ 63 3 

♦ 

+ + + + + + 



♦♦♦♦♦♦♦♦♦♦ 
♦ 

+jf~ 66 ♦ 



♦ 

'♦ 
♦ 
♦ 

4 



IS 
3 



♦ 



• 955 
1.000 
1.000 



• 955 

• 833 
1«0Q0 



Llghtts 
Extens | on 
r o* Kappa 



.818 
1,000 

llooo 



Category Marginal Agreement I .929 
Cohen" s Kappa i .904 



ERIC 



530 



501 



v 



Table '260 m ( 

Phas/»OIV - IGE Evaluation Project 
Descr I pt I v« Study • SPMnfl 1978 

Re I latil .1 | tv on Merer I a I • 
Observer 40 • Expert 80 
'school 801 Grade 2 
week 9 Day 5' 
OMp - 



* Meter la*! I PaPer/pencM 



P 



Category chosen 
by Expert 80 



Used , 
Not used ' 
r other 

Mar 01 na I 



Cat. 
# 



Cateaory chosen 
by Observer 40 



12 3 Totals 



1 


♦ 
♦ 


66 


0 


0 


♦ 


66 ♦ 


1.000 


.971 


1.000 


2 


♦ 


2 


13 


0 


♦ ■ 
♦ 


IS ♦ 


.867 


.867 


,842 


3 


♦ 


0 


0 


3 


♦ 




1 .000 


1.000 


1.000 




♦ 








♦ 




* 






♦♦♦♦♦♦♦♦♦ 






♦ 










Totals 


♦ 


68 


13 


3 


♦ 


84- ♦ 


i ' ' ■ 







Ll0ht»S 

Extens ion 
Agreement Agreement o* Kappa 



Marginal Nominal Marginal 



category Marginal Agreementi 
Cohen**' Keppal , .929 



.946 



ERIC 



V 



Table 261 

> 

Phase IV - IGE Evaluation Project 
OeicriPtlve Study - Spring 1978 

% Re I | ab I 1 1 ty ( on Materials 
Observer**^. Export 90* 
School 601 Grade S 
Week 9 Day 4 
DMP • 



Material I Manipulative 



Category choCen 
^by Expert 90 



Used 

Not used 
Other 



Cat v 



♦ 

1 ♦ 
♦ 

2 ♦ 
♦ 

3 ♦ 

. ♦ 



Cateohry chosen 
by Observer 10 



Mara I na I 
' Total s 



0 

1 



4 
BO 
1 



0 
0 



♦ 
♦ 
♦ 
♦ 
♦ 
♦ 



4 
61 
5 



+++++++++++++ 
♦ ♦ 



Marginal Total s ♦ 

♦ 



1 85 



♦ 



90 



fiom | na I 
Agreement 



0.000 

•Vea 



eioo 



Marginal 
Agreement 



• 250 
.953 

• 800 



L^ght *S 
Extens | on 
of Kappa 



• 011 

• 778 

• 7.91 



504 



category Marginal Agreement! »668 
Cohen »s Kappa i ,5*7 



505 



Table 262 



Phase IV • I6E Evaluation Project 
Descriptive Study - Spring X978 

Rel lab'i I I ty on Mater I a I s 
Observer 20 Expert 90 
School 801 Grade 5 
Week 9 Day' 4 
DMP 



Material I Manipulative 



Category chosen 
by expert 90 



used 

Not used 
Other 



Cat. 



Category chosen 
by Observer 20 



506 



♦ ♦<« 

1 + 

2 ♦ 
♦ 

3 ♦ 
♦ 



Marginal Total* 



♦ 



1 

0 
0 
0 



4 
79 
0 

83 



0 
2 
5 



Marg I na I 
Total s 



♦ 
♦ 

4 
♦ 



4 
81 
5 

90 



Nominal 
Agreement 



0.000 
.975 
1.000 



Marginal 
Agreement 



0.000 
,976 
,714 



cateoory Margma' Agreement! ,563 
Cohen's Keppet ,598 



Llghtis 
Extens Ion 
of Kappa 



0,000 
,683 
1,000 



. Table 263 

Phase IV - tQ€ evaluation Project 
Descriptive Study - spring 1978 

Re I lab 1 1 1 tv on Material • . 
Observer ^0 Expert 80„ K 
School 801 Grade 5 
Week 9 Day A 
DMP 



Materlali Manipulative 



/ 



Category chosen 
by Expert 80 



Used 

Not used 
Other 

\ 

\ 

\ 



Cat, 
ft 



Category chosen 
by Observer 10 



1 ' 


2 






is 


0 






1,8 


45 


4 


2 



1 ♦ 

♦ 

2 «■ 

3 ♦ 
♦ 

Marginal Totals ♦ 37 

♦ 



47 



0 
0 
6 

6 



MarOl nal 
TotaJ s 



Nomina I 
Agreement 



Llghtis 
Marginal Extension 
Agreement of Keppa^ 



^ 15 ♦ l»0O0 

♦ ' ♦ x 

♦ 63- .» 
♦ 

♦ 12 ♦ #500 

♦ ♦ 

4 ♦ 

♦ 90 ♦ 

♦ ♦ 



• 405 

• 746 
,500 



1,000 

• 402 

• 464 



9 

ERIC 



508 



category Marginal Agreement! ,550 
c©hsn»s Kappist ,521 



509 



Table 264 



\ Phase IV - IGE Evaluation Project 
Descriptive Study - Spring 1978 



Re l | ab I I I ty on 
Observer ^jO 
School 601 
Week 9 



Mater I a I s 
Expert 80 
'Grade 2 / 
Day 4 



DMP 



ON 



\ 



Material I Manipulative 



Category chosen 
by Expert 80 



Used 
/■■Not, used 
Other 



Cat • 
M 



Category chosen 
by Observer 10 



1 



51 0 



1 + 

2 ♦ 
♦ 

3 ♦ 



Marginal Totals ♦ 

♦ 



0 
1 
0 



9 
64 
3 

76 



0 
0 
1 



Marginal Nominal Marginal 
Totals ' Agreement Agreement 




\ 



♦ 

4 


9 


♦ 


OtOOO 


• 111 




65 




#985 


, .BS5 


♦ 
♦ 


4 




• 250 


• 250 












♦ 










♦ 




♦ 






♦ 




♦ 







Category Marginal Agreement I »405 
Cohen's Keppal «103 



LI9htt§ 
Extension 
of Kappa 



-♦013 
\ ,400 



♦ 240 



9 

ERIC 



Table 265 " 



Phase IV - XGE Evaluation Project 
Descriptive Study - spring 1978 



Re I lab 1 1 1 ty on 
Observer 10 
School 801 
Week 9 

OMP 



Mater la! • 
Expert 90 
Grade S 
Day 5 



Mater I a I l Menipulat Iva 



t 



Category chosen 
by Expert 90 



Cat. 
n 



Category chosen 
by Observer 10 



1 



Marg I na I 
Tota I s • 



Nom | na i 
Agreement 



Light* s 
Marginal Extension 
Agreement of Kappa' 



Used 

Not used 
Other 



1 

2 
3 



Marginal Totals 



i 

0 0 

2 75 

0 2 



77 
N 



0 
0 
11 

11 



♦ 
♦ 

♦ 
♦ 
♦ 



0 
77 
13 

90 



♦ 

♦ 
♦ 

♦ 
♦ 



• 97* 

• 846 



0*000 
1.000 
• 846 



• 820 

• 825 



Cohen's KappaJ .822 

512 • . 



Category Marginal Agreements «615 

. < 513 



Table 266 



Phase IV "IQE Evaluation Project 
Descriptive Study - spring 19T8 

Rel labl I Ity on Material* 

Observer 30 Xxpert 80 ' 
. Schook'801 Grade 5 
Week 9 Day 5 
DMP 



H 

or 



Materiel I Man | pu I at I v© 



Category chosen 
by Expert 80 



Cat. 



Category chosen 
by Observer 30 



MarQ I nal 
Total s 



Nom | na I 
Agreement 



Marginal 
Agreement 



Ulght»s 
Extension 
of Kappa 



Use^ 
Not used 
Other 



♦ 

a ♦ 
♦ 

3 ♦ 
♦ 



10 2 
Ik 62 
1 1 



Marginal' totals ♦ '25 

♦ 



'65 



0 
0 
0 



+ 

♦ 
♦ 
♦ 
♦ 













♦ 


• B33 


• 400 










76 




• 816 


.855 




♦ 






2 




0*000 


0*000 


















90 










♦ 


1 » 





,T69 
,337 
0,000 



514 



Category Marginal Agreementl .445 

* ' 

Cohen's Kappal .434 



Category chosen 
by Expert 80 



Used 

Not used 
Other 



f 



Table 267 



I 



Phase !V - IflE Evaluation Project 
Descriptive study - spring 1978 



Rel |Abl I Ity on Materials 
Observer AO Expert Bo 
School 801 Grade 5 
Week 9 Day 5 
OMP 



/ 

Mater I aj t Han I pu I at lye 



Cat 
* 



CateQ^ry chosen 
by Observer <»0 

1 2 3 . 



Marginal 
Total s 




Nom f ; ne I 
Agreement 



Marginal 
Agreement 



Marginal 




♦♦♦♦♦♦f ♦♦♦♦♦ 



Light »s 
Extens Ion 
of ippa 



12 


♦ 


/ i583 


1 

• 667 


..543 


76 ' 


♦ / 




• 950 


• 882 


2 


+ 

4 


1 #000 


1.000 


1.000 


90/ 











X 



516 



9 

ERIC 



category Marginal Agr/aernenti 
Cohen* s Kappal «719 



.872 



H 



517 



Table 268 



Phase IV - IGE Evaluation Project 
Descriptive! Study - spring 1978 

Re I I ab 1 1 Itv on Materials 
Observed 30 Expert BO 
School B01 Gl*ade 2 ^ 
WeeK\ 9 '.D«fr- 5 
N WORSD „ 



O 



i 



Cate<?oryNshosen 
by Expert 80 



ysed 

4 



51b 



Not used 
other 



Mater J aU Man | pu I at J ve 



\ 



Cateoory chosen 
' by Observer 30 



Cat. 



Marol nal 
' Tota.l s 



















+ 






♦ 


56 


♦ 


1 


♦ 53 


3 


0 




♦ 










♦ 






2 


♦ 0 


0 


0 


♦ 


0 


♦ 
















3 


+ 2 


0 


2 


.♦ 


4 






♦ 






♦ 




4* 



Nom | na I 
Agreement 



• 946 



Marginal Totals ♦ 

♦ 




category Marginal Agreemehtl 
Cohen's Keppal .414 



• 500 



Marginal 
Agreement 



,962 
0,000 
• 500 



• 494 



LI ght is 
EXtens ion 
of Kappa 



,357 



,483 



519 



ERIC 



/ 



Table 269 

Phase IV - !GE Evaluation Project 
Descriptive Study - spring 1978 

Rei I ah I I I ty on Materials 
Observer 40 Expert Bo 
School 801 Grade 2 
Week 9 Day 5 
WDRSD 



Materiall Manipulative 



Category chosen 
by Expert 80 



Cat » 
n 



Category chosen 
by Observer 40 



1 



ware I r a I 
Total s 



Nom | na I 
Agreement 



LlOhtfS 
Marginal Extension 
Agreement of Kappa 



used 

Not used 
Other 



Marginal 





♦ 












1 


4 45 


11 


0 


4 


56 


4 










♦ 






? 


♦ 0 


0 


0 


♦ 


0 






♦ 






♦ 






3 


4 0 


2 


2 


♦ 


4 






♦ 






♦ 








444444444 4444 






















4 


* 


4 45 


13 


2 




60 


4 




4 






4 




4 



• 804 



• 500 



,804 »214 

0 t 000 »— « , 

• 500 ^83 



520 



category Marginal Agreement! 
Cohen's Kappaf »272 



• 435 



521 



Table /270 

Phase IV - IGE Evaluation Project £ 
Descriptive Study - SPrlnfl 1978 £ 



Re I | eb I I! ty on Mater lal s 
Observer 10 Expert 90 
School 801 Grade 2 
Week 9 Day 5 
DMP 



Mffterlall Manipulative 



Category chosen 
by Expert 90 



Used 

Not used 
Other 



Cat, 



Cateaory chosen 
by Observer 10 



*>2 



Maro I n» I 



1 ♦ 0 0 
♦ 

2 ♦ 1 73 
♦ 

3 ♦ 0 3 
♦ 

Totals ♦ 1 76 
♦ 



f 



0 
3 
4 



Marolne 
Tote I s 



Nom | na I 
Agreement 



♦ 
♦ 

♦ 

♦ 

♦ 
♦ 



o 

77 
7 

84 



♦ 
♦ 
♦ 
♦ 
♦ 

+ 

♦ 



• 948 



\ 



Marginal 
Agreement 



0*000 
#987 
1,000 



Llght«s 
Extens I on 
of Kappa 



,455 
,532 



523 



category Marginal^ Agreementl ,662 



Cohen's Keppal ,491 



I8» 



r-, 



Tatxle 271 

Phase IV- IOE Evaluation Proj«ct 
Descriptive Study - spring 1978 

Re I lablV I ty on Material! 
. Observer' 30 Expert Bo 
School 801 Grade 2 
Week 9 ' Day 5 
DMP 



Material I Manipulative 



Category chosen 
by Expert 80 



Cat* 



Cateaory chosen 
by Observer 30 



4 

Marol nal 



Nominal Marginal 



Uohtti 
Extens I on 



arg na wominoi - - — - 

Totals^ Agreement Agreement of Kappa 



used • 
Not used 
Other 



♦ 

1 ♦ 

?. ♦ 
♦ 

3 ♦ 
♦ 



0 
0 
0 



Marginal Totals ♦ 

♦ 

♦ ♦♦♦ 



1 

80 
0 

81 



0 


♦ 


1 


♦ 


Otftoo 


Of 000 


0,000 


0 


♦ 
♦ 


60 




uooo 


• 9B8 


1,000 


3 




3 


♦ 
♦ 


uooo 


UOOO 


l,0frO 




♦ 


♦ ♦ ♦ ♦ 












♦ 








* 




3 




84 












♦ 













category Merslnal Aoreament I .663 
eohan'f Kappal .888 



5 £4 



525 



Table 272 

Phase TV - l«£ Evaluation Project 
Descriptive Study - SPrlno. 1978 

pel |abl I I ty on Maters alt 
Observer 40 Expe/t So 
School 801 <5ra4e 2 

Week 9 Day 5 V 
OMP 



Material I Manipulative 



Category chosen 
by Expert BO 



Cat * 
* 



Category chosen 
by Observer 40 



1 



Marglna 
. fota I s 



Nom I na I 
Agreement 



MarOl nal 
Agreement 



Light • s 
Extens I on 
of Kappa 



52U 



Used 

Not used 
other 



♦ 

1 ♦ 
♦ 

2 ♦ 
♦ 

3 



1 
0 



: / 



0 
80 
0 



Marg I na I 



Totals ♦ 1 80 
♦ 



0 
0 
3 



♦ 
♦ 

♦ 
♦ 
♦ 
♦ 



1 
80 
3 

84 



♦ 

♦ 
♦ 

♦ 



1.000 
1.000 
1*000 



category MarOinal Agreementl 1.000 
Cohan's Kappef 1.000 



1.000 
1.000 
1.000 



1.000 
1.000 
1.000 



527 



9 

ERIC 



Table 273 

Phase IV - IGE Evaluation Project 
Descriptive Study - Spring 1978 



Re I lab I I I ty on 
Observer AO 
School 801 
Week 9 

DMP 



Mater I el s 
Expert Bo 
Grade 5 
Day 4 



Materiel I Game 



Category chosen 
by Expert 80 



Catt 
* 



Category chosen 
by Observer 10 



1 



'3 



Maroi nel 
Totals 



Nomine I 
Agreement 



Marginal 
Agreement 



Light* s 
Extens Ion 
6f Kappa 















♦ 






i 






Used 


1 


♦ 


0 


4 


0 




4 


♦ 


0*000 


o.boo 


0*000 






+ 








♦ 




♦ 








Not used 


? 


♦ 


0 


74 


0 




74 


♦ 


1*000 


'•681 


1,000 






♦ 








♦ 




♦ 








Other 


3 


♦ 


.0 


6 


6 




12 


♦ 


• 500 


• 500 


• 464 






♦ 








♦ . 




♦ 









Marginal Totals ♦ 

4 



84 



♦ 



90 



528 



Category Marginal Agreementi »460 

Cohen»s Keppal ,503 529 



/ * 

/ Table 274 

Phase !V - IGE Evaluation Project 
Descriptive Study - spring 1978 

pel lab til ty on Materials" 
/ Observer lo Expert 90 
/ School 801 Grade 2 

Week 9 Day 5 
DMP 



/ 



V?"- 



Category .chosen 
by Expert.- 90 



used 
Not used 
Other 



Mater (al I Game 



Catt 
0 



530 



Category chosen 
by Observer 10 

12 3 
♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 



HarQl nal 
. Totals 

♦♦♦♦♦♦♦♦♦ 



Nominal 
Agreement 



Marginal 
Agreement 



Vlght t • 
Extens ion 
of Kappa 



1 


♦ 
♦ 


* 0 


33 


1 


♦ 34 


■ ♦ 


OtOOO 


0.000 


0,000 


2 


♦ 
♦ 


0 


41 


2 


♦ 43 




• 953 


• 558 


,442 


3 


♦ 


0 


3 


4 


♦._ 7 


♦ 
♦ 


• 571 


uooo 


• 532 




♦ 










♦ 


























Total s 


♦ 
♦ 


0 


77 


7 














* 










♦ 









♦♦♦♦♦♦♦♦♦ 



531 



category Marginal Agreement! «519 



Coheres Kappa; 



114 



Table 275 

) 

Phase XV - IGE Eval 
Descriptive Study 



**5> 

Re I ; ab 1 1 1 ty on 
Observer 20 
School 601 
Week -9 

DMP 



uatlon Project 
- spring 1978 




Mater la I s 
Expert 90 
Grade 2 
Day 5 



Hater raj I Game 



Category chosen 
by Expert 96 



used 

*Not used 
Other 



Cat* 



Category chosen 
by Observer 20 

12 3 



14' 0 33 
g ♦ 0 *0 

3 ♦ 0 2 
♦ 

♦♦♦♦♦♦♦♦♦♦ 
Marginal Totals ♦ . 0 75 



iS 

3 

5 



Maroinal Nominal 
Totals Agreement 



Marginal 
Agreement 



Llght»s 
Extens | on 
of Kappa 



♦ 



43 



64 



♦ 


0.000 


0*000 


|»000 




• 930 


,573 


• 349 


♦ 




•<T76 


„ ,680 




































532 



category Marginal Agreementl #450 
Cohen's Kappal #131 



533 



9 

ERIC 



Table 276 

Phase IV - I«E Evaluation Project 
Descriptive Study - SPrlnO 1978 ^ 

Re I I ab I I I tv on Materials 
Observer 30 Expert 80 
School 801 Grade 2/ 
Wee k 9 Day S 
DMP 



Mater I 3 1 1 Game 



Category chosen 
by expert 60 



Used 

Not used 
Other 

Mara I na I 



534 



Cat , 
* 



Cateqory chosen 
by Observer 30 



1 ♦ 8 0 

2 ♦ 2 71 

3 ♦ 0 0 

♦ 

* ota I s ♦ 10 71 
♦ 



0 
0 
3 



Mero I na 
Total s 



Noml na I 
Agreement 



♦ 
♦ 
♦ 
♦ 
♦ 
♦ 
♦ 

♦ 



8 
73 

3 

84 



♦ 

♦ 
♦ 
♦ 
♦ 

♦ 



1.000 
.973 
1.000 



category Meroinet Agreement* .924 
Cohen's Kappat .906 



Marolnai 
Agreement 



,600 
,973 
1.000 



Light IS 
Extens I on 
of Kappa 



1,000 
,823 
1,000 



535 



Tab ly*7 7 

Phase IV - X6E Evaluation Projact , 
Descriptive Study - spring 1978 X 

Re I lab! I Ity on Materials 
Observer 40 Expert 80 
School 801 Grade 2 
Week 9 Day 5 
DMP 



Mater lal t Game 



Category chosen 
by Expert 80 



Cat • 



Category chosen 
by Observer uo 



1 



Marginal 
Teta |s 




f 

Noml na I 
Agreement 



.750 
1.000 
1.000 



MargJ<na I 
AgreW*n>nt 



.750 
.973 
1.000 



UIOHttS 
Extens I on 
o* Kappa 



.731 
1.000 
1.000 



536 



Category Marginal Agreement! 
Cohen's Keppal .890 



.908 



537 



Table 278 

phase IV - 1GE Evaluation Project 
Descriptive Study - spring 1978 

Re I | ab I I I ty on nater I a I s 
Observer 20 Expert 9o 
School 801 Grade 2 
Week 9 Day 5 
WDRSD 

4 

Material! Audio/visual aid 



Category chosen 
by Expert 90 

Used 

Not used 
Other . 



Category chosen 
■*bv Observer 2o 



Cat. 



1 ♦ 
♦ 

3 ♦ 



0 0 
3 55 
0 0 



0 
0 
8 



Marttl na 
Tota I s 



Light»s 

Nomina! Marginal Extension 
Agreement Agreement of Kappa 



♦ 
♦ 
♦ 
♦ 

+ 



0 
56 
8 



♦ 
+ 



• 948 

1*000 



0.000 
.948, 
1.000 



.690 
1 .000 



533 



Marginal Total* 



ss 



8 



66 



530 



Category Marginal Agreementl .649 
Cohen ♦* Kappa I i>820 



Table 279 



Phase IV - IGE Evaluation Project 
Descriptive Study - SPrlnO 19T8 

Reliability on Materials 
ObserverNkO Expert 9o 
School 8oY Grade 5 
Week 9 Day 4 
OMP 



Material! Printed material 



Category chosen 
by Observer 10 



Category chosen 
by Expert 90 


Cat. 
u 


i 


2 


3 


Used 


1 ♦ 


e 


3 


0 


Not used 


a <• 


2 


72 


0 


Other 


3 ♦ 


1 


0 


4 



Marginal Totals 



11 75 



4 , 



Marg I na 
Totals 



♦ 
♦ 

♦ 

♦ 
♦ 
♦ 



11 

74 
5 

90 



Norn I na I 
Agreement 



• 727 
,973 

• 600 



Marginal 
Agreement 



l.ooo 

• 987 

• 800 



Llght«s 
Extens I on 
of Kappa 



,689 

• 838 

♦ ?91 



Category Marginal Agreementl «929 
Cohen's Kappat «776 



510 



541 



Table. 280 

Phase IV - IOC Evaluation Project 
Descriptive Study - Spring 1978 

Ret labl I I ty or, materials 
' observer 20 Expert 90 
School 801 Grade. 5 
V Week 9 Day k 
DMP 



to 



Meterlall Printed materia! 



Category chosen 
by Expert 90 



used 

Not used 



512 



Other 



Cats 
* 



1 

2 
3 



Marginal Totals 



Category chosen 
by "Observer 20 



7 
1 
0 



t 



71 
0 



6 75 



0 
2 
5 



Marol nal 
Total s 



11 
74 

S 

90 

♦♦♦♦♦♦♦♦ 



LlPht»S 

Nominal Marginal Extension 
Agreement Agreement of Kappa 



• 636 

• 959 
1*000 



.4. 



category Marginal Agreement! .809 

* 

Cohen's Kdppat .7*0 



.727 
,987 
• 714 



• 601 
,757 
1,000 



543 



~> ' Table 281 



Phase TV - IOE evaluation Project 
Descriptive Study - SprlnO 1978 

R«l |abl I Ity on materials 
Observer *jQ ' Expert 80 
School 801 Grade S 
week 9 Day 4 
DMP 



r 



i/ 



Category chosen 
by Expert 80 



Cat* 



Material! Printed material 



Category chosen 
by Observer 10 



MarQ I na 
Total s 



Now I na I 
Agreement 



Marginal 
Agreement 



LlgKt»s 
Extens | on 
of Kappa 



Used 

Not used 
Other 



1 ♦ 
♦ 

2 ♦ 
♦ 

3 ♦ 
♦ 



Marginal Totals «■ 



14 

5 

0 

19 



0 
59 
6 

65 



0 
0 
6 



♦ 
♦ 
♦ 
♦ 
♦ 
♦ 

♦ 
♦ 



14 
64 
12 

90 



♦ 

4 
♦ 
♦ 
♦ 

♦ 

♦ 
♦ 
♦ 



1*000 

• 922 

• 500 



•737 1.000 
•98S »719 
,500 ^64 



544 



Category Marginal Agreementi 
Cohen's Kappai .725 



• 740 



515 



Table 282 

phase TV - IOE Evaluation Project 
Descriptive Study - Spring 1978 

Rei lab) II ty on Mater lal 9 
Observer 40 Expert BO 
School 801 Grade 2 
Week 9 Day 4 
WDRSD 



V 



MateriaM Printed 



mater I a I 



<*1 



Category chosen 
by Expert 80 



Used 

Not used 
Other 



Cat. 



Category chosen 
by Observer 40 



1 



* 



1 ♦ 
♦ 

2 ♦ 
♦ 

3 * 



8 


9 


0 




17 


0 


25 


0 * 


♦ 


25 










0 


0 


18 


♦ 


18 



Marginal 
Total s •> 

♦ 
* 
♦ 

♦ 



Nom ( na I 
Agreement 



,471 

1 ,000 
1,000 



Marginal 
Agreement 



,471 
,735 

'1,000 



1 1 ght ♦ s 
Extens l on 
of Kappa 



,389 
1,000 
1,000 



54U 



8 



34 



Marginal Totals ♦ 

4, + + + + + + * + * ♦ * • 

t 



18 



♦ 



60 



♦ 



547 



Category Marginal Agreement! ,735 



Cohen's Kapoa? ,764 



'Table 283 

Phase IV - Iq\ Evaluation Project 
Descriptive ^tudy - spring 1978 

Re I lab I I Ifcy on Materials 
Observer Expert 80 

School 801 Grade 2 
Week ,9 Day 4 
DMP 



Mater I a I l 



/ 

Pcrtnt 



nted material 



Category chosen 
by Expert 80 



Cat, 
H 



Category chosen 
by Observer 10 



Maro I na 
Tota I s 



Norn | na I 
Agreement 



Mars | na I 
Agreement 



Llght»s 
Extens i on 
of Kappa 



used 

Not used 
Other 



1 

2 
3 



Marginal Totals 



♦ 
♦ 
♦ 
♦ 
♦ 

♦ 

+ 

♦ 



1 

65 
3 

69 



0 
0 
1 



♦ 
♦ 


7 


♦ 
♦ 


• 857 


• 875 


• 841 


♦ 
♦ 


67 


♦ 
♦ 


• 970 


• 971 


,741 


♦ 


4 


♦ 


• 250 


• 250 


• 240 






♦ 




















♦ 


78 














♦ 









o 

ERIC 



548 



category Marginal Agreementi 
Cohen's Kappa I .666 



699 



v'l 



54'J 



/ 



Table 284 



Phase IV - I6E Evaluation Project 
Descriptive Study - spring 1978 > <g 

Reliability on Mater I a I s 
Observer 10 Expert 9o - 
School 801 Grade 5 
week 9 Day 5 
DMP 



Material! Printed material 



Cateqory chosen 
by Observer 10 



Category chosen 
by Expert 90 



Cat, 
* 



Maro I na 
Tota I s 



550 



1 2 3 
+ + + + > + + + + + + + + + <'• + ♦+♦♦ + ♦♦♦ + + ♦♦♦ 



Nom | na I 
Agreement 



Marg | na I 
Agreement 



♦ ♦ ♦ <♦ 

Category Marginal Agreement! .618 
Cohen*s Kappa! ,567 



LlOhttf 
E* tens I on 
of Kappa 















♦ 












Used 


1 


♦ 


0 


1 


0 




1 




0*000 


.100 


-•125 






♦ 








♦ 












Not used 


? 




10 


66 


0 








#866 


• 908 


,436 














♦ 












Otber 


3 


♦ 


0 


2 


11 


♦ 


13 


♦ 


#846 


• 846 


,825 






♦ 








4 




































Mara I na 1 


Tota 1 s 




10 


69 


11 


♦ 


90 



































9 

ERIC 



< 



Table 285 

Phase IV - I(*E Evaluation Project 
Descriptive Study - SPrlno 1978 



Re | | ab I 1 1 ty on Material s 
Observer 20 Expert 90 
School 801 Grade 5 
Week 9 Day 5 
DMP 



Material! Printed material 



Category chosen 
by Expert 90 



Cat » 



Category chosen 
by Observer 20 



Maro I na 
Total s 



Nom | na I 
Agreemen t 



Marg I na I ' 
Agreement 



L|ght»s 
Extens | on 
of Kappa 















4 








Used 


1 




0 


1 


0 




1 


♦ 


0,000 


















♦ 




Not used 


1 2 


♦ 


0 , 


76 


0 




76 




It 000 






♦ 












* 




Other 


3' 




0 


1 


18 




13 




• 923 






♦ 

















0.000 0,000 
,974 1,000 
•923 .911 



Marginal Totals •♦• 

♦ 

\ 



78 



12 



♦ 



90 



♦ 
♦ 



category Marginal Agreement! #632 
Cohen's Kappat ♦ 91 1 



-0 



r r; ' ) 
O O ^ 



553 



Table 286 

Phase IV - IGE Evaluation Project 
Oescriotl /e Study - spring 1078 . £ . 

00 

pel I Hb I I I ty on Mater I a I s 
Observer 30 Expert 8o 
School 801 Grade 5 
Week 9 Day 5 
DMP 



Material! Printed material 



9 

ERIC 



Category chosen 
by Expert 80 



Cat • 
# 



Category chosen 
by Observer 30 



1 



Marg I r,a I 
Tota I s 



Nom | na I 
Agreement 



Marolnal 
Agreement 



L|ght»s 
Extens | on 
of Kappa 



Used 

Not used 

"other 



1 

2 
3 



13 
2 



^ A 
J J 1 



Marginal Total • 



♦ 
♦ 

♦ 15 
+ 



\ 



1 

72 
2 

75 



0 
0 
0 



♦ 

♦ 
♦ 

♦ 



14 
74 

2 

90 



♦ 
♦ 



• 929 

• 973 
0.000 



• 933 
.987 
OtOOO 



.914 
• 636 
0.000 



550 



category Marginal Agreement l t6<*0 
Cohen's Kappa* ,808 



Table 287 



Phase IV - XGE Evaluation Project 
Descriptive Study - spring 1978 

Re I I ab 1 1 I tv on Mater I a 1 1 
Observer 40 Expert 80 
School 801 Grade S 
Week 9 Day 5 

OMP 



Material! Printed material 



Category chosen 
by Expert 80 



Cat, 



Cateaory chosen 
by Observer 40 



1 



Marginal Nominal Marginal 
Totals Agreement Agreement 



Llght»s 
Extens I on 
of Kappa 



♦ 




♦ 














Used 


1 


♦ 


12 


2 


0 


♦ 
♦ 


14 


♦ 


Not used 


2 


♦ 
♦ 


1 


73 


0 




74 


♦ 


Other 


3 


♦ 


0 


0 


2 


4 


2 


♦ 



Marginal Totals * 

♦ 




,857 
.986 



1.000/ 



.929 
.987 
1.000 



.833 
.919 
1,000 



550 



9 

ERIC 



Category Marginal Agreement! .972 
Cohen*"* Kappal .886 



557 



i 



Table 288 

Phase IV - IOE Evaluation Project 
Descriptive SJJtl^v " Spring 1978 

Re I | ab I I I XV on Mater I a I s 
Observer 10 Expert 9o 
■ School 801 Grade 2 

week 9 Day 5 
WDKSD 



o 



Materiel! Printed material 



Category chosen 
by Expert 90 



Cat, 
u 



Category chosen 
by Observer 10 



1 



Mars I na I 
Tota I s 



Nom | na I 
Agreement 



Marg I na I 
Agreement 



Light • s 
Extens | or 
of Kappa 



558 







♦ 








♦ 




♦ 








Used 


1 


♦ 


10 


4 


0 


♦ 


14 


♦ 


• 714 


.714 


• 663 


























Not used 


2 




0 


43 


1 


♦ 


44 


♦ 


' ,977 


• 936 


• 921 






♦ 








♦ 




♦ 








Other 


3 




0 


0 


6 


♦ 


8 


+ 


1.000 


• 889 


1.000 






♦ 








♦ 













Marginal Totalt ♦ 10 



47 



♦ 



66 



♦ 
♦ 



559 



category Marginal Agreementl .846 
Cohen's Kappal .841 



ERIC 



Table 289 

Phase IV - IOE Evaluation Project 
Descriptive study - spring 1978 

Pol labl I I ty on Materials 
jObserver 20 Expert 90 
School 801 Grade a 
' week 9 Day 5 \ 
WDRSD 



Matorlali Printed material 



Category chosen 
by Expert 90 



Cat . 



Category chosen 
by Observer 2o 



/ 



Marginal Nomina! Marginal 
Totals Agreement 



U | ght i s 
Extens | on 
Agreement . Kappa 



Used 

Not used 
Other 



♦ 

r 

1 ♦ 

/ ♦ 

2 ♦ 
+ 

3 ♦ 
♦ 



0 
0 
0 



14 



44 



Marginal Totals + 



58 



0 
0 
8 



♦ 



14 
44 
6 

66 



♦ 

♦ 

♦ 
♦ 



0,000 
1,000 
1,000 



0,000 
,759 
1,000 



0,000 
1.000 
1,000 



560 



category Marginal Agreement; ,586 
Cohen's Kappal ,469 



5G1 



\ 



Table 290 

Phase TV - I*E Evaluation Project 
Descriptive Study - SPrlnQ 1978 

Re I I ab I I I ty on Mater I a I s 
Observer 10 Expert 90 
School 801 Oracle 2 
Week 9 Day 5 
DMP 



Material) Printed materiel 



Category chosen 
by Expert 90 



Cat* 
U 



Category chosen 
by Observer 10 



1 



Marg I na I 
Tota I s 



Norn I na I 
Agreement 



Merg I na I 
Agreement 



Light «s 
fcxtens | on 
of Kappa 



Used 

Not used 
Other 

















♦ 




1 




0 


1 


0 




1 


♦ 


0 *000 




















2 


♦ 


0 


73 


3 




76 


> 


• 961 




















3 




0 


3 


4 




7 




• 571 




♦ 












♦ 





Marginal Totals ♦ 



\) b t* 



77 



♦ 



84 



♦ 
♦ 
♦ 



0.000 
.967 
1.000 



0,000 
,52fc 
,53a 



/ 



56 



category Marginal Agreements .662 
Cohen's Kappal .491 



Table 291 



X 



Phase IV - l«Z Evaluation Project 
Descriptive Study - spring 1978 

Re 1 1 ab I 1 1 1 y Mater lal s 
Observer 20 Expert 90 
School 801 Grade 2 
week 9 Day 5 
DMP 



Material! Printed material 



Category chosen 
by Expert 90 



used 

Not used 
Other 

Mara | na 



Cat. 



Category chosen 
by Observer 20 



1 



♦ 

1 ♦ 0 1 
♦ 

? ♦ 0 72 
+ 

3 ♦ 0 2 

■* 

♦ 

Totals ♦ 0 75 
♦ 



0 
5 



Maro I na I 
Total s 



+ 

♦ 
♦ 

* 

* 
♦ 



1 

76 
7 

6** 



♦ 
♦ 

♦ 
♦ 
♦ 
♦ 

•f 

♦ 



Nom i na I 
Agreement 



0,000 
,947 
.71* 



Maro I na I 
Agreement 



0,000 
,987 
.778 



Ulght»S 
Extens ion 
of Kappa 



0,000 
,509 
,680 



Category Marginal Agreementl ,588 
Coh«n»s Kappal .5*5 



5 (J 5 



Table 292 



Phase IV - XOE Evaluation Project 
Descriptive Study - spring 1978 

Reliability on Material! 
Observer 10 Expert 9 a 
School 001 Grade 5 
Week 9 Day 4 
DMP 



4> 



*ater I a I I Othpr 



Category chosen 
by Expert 90 



Cat. 



Category chosen 
by O^sjsrver 10 



List's 



«) o u 



Used 


1 


4 


11 1 






4 




Not used 


2 


♦ 


C. 73 






4 




OV >er 


3 


♦ 


0 1 






4 








444S 


f 4 4 4 4 4 4 










Mara I na 1 


Tota I S 




11 75 






4 


< 



♦4444444 4 



0 
0 
4 



MarO 1 na 




Nom | na l 


Maro | na J 


Extens 1 on 


Total s 




Agreement 


Agreement 


of Kappa 


4 


4 








4 12 


4 


• 917 


.917 


,905 


4 


4 








4 73 


4 


, 1.000 


,973 


1.000 


4 


4 








4 5 


4 


.800 


• 800 


,791 


4 


4 










567 



category Marginal Hgreemehtl .897 
Cohen's Kappat ,927 



9 

ERIC 



Table 293 

Phase IV - IOE Evaluation Project 
Descriptive Study - spring 1978 



Reliability on 
Observer 20 
School 801 
week 9 

OMP 



Kater I a I s 
Expert 90 
Grade 5 
Day 4 



Materlall Other 



Category chosen 
by Observer 20 



Category chosen 


Cat . 










by Expert 90 


* 




. 1 


2 


3 


Used 


1 


* 
♦ 


11 


1 


0 


Not used 


2 


♦ 
♦ 


2 


69 


2 


Other 


3 


♦ 


0 


0 


5 



Marginal Total* ♦ 

♦ 



13 



70 



Marg I na I 
Total • 



Nomina I 
Agreement 



Marginal 
Agreement 



♦ 

4 
♦ 

♦ 
♦ 
♦ 











12 




,917- 


• 923 




♦ 






73 


♦ 


• 945 


.959 










5 


♦ 


1.000 


.714 




♦ 














90 


♦ 








♦ 







Llghtts 
Extens I on 
of Kappa 



,903 
,753 

1,000 



9 

ERIC 



category Marginal Agreementl ,865 
Cohan's Kappal .839 



5GB 



569 



LI 



Table 294 

Phase IV - I6E Evaluation Project 
Descriptive Study - Spring 1978 

Re I lab I II tv on Materials 
Observer ^0 Expert 80 
School 801 Grade 2 
Week 9 Oay k 
OMP 



Material! Other 



Category chosen 
by Expert 80 



used 

Not used 
Other 



Cateaory chosen 
by Observer 10 



Cat » 
* 



1 ♦ 

2 ♦ 
♦ 

3 + 

4 



1 2 

2 



0 
0 



570 



Marginal Totals ♦ 

♦ 



0 
72 
3 

75 



0 
0 
1 



Marginal /" Nominal 
Totals Agreement 



+ 
♦ 
♦ 



2 
72 
<♦ 

78 



♦ 
♦ 
♦ 
♦ 
♦ 

♦ 
♦ 



1.000 
1.000 
.250 



Marginal 



- Llght»s 
Extens ion 



Agreement of Kappa 



1 .000 
• 960 
.250 



1 ,000 

i .ooo 

.2^0 



5 



Category Mftrgmal Agreement!* 
Cohen's Kappa! »6S> 



,737 



Table 295 



Phase IV - XOE Evaluation Project 
Descriptive Study - soring 1978 

Rel labl Mty on Material* 
/ Observer iO Expert 90 

School 801 Grade 5 
Week 9 Day S 
DMr» 



Material! Other 



Category chosen 
by Expert 90 



used 

Not VJ 
Other 



Cat 



1 
2 

3 



Category chosen 
by Observer 10 



1 



Marginal Total* 



♦ 
♦ 

♦ 



2 
0 



71 

2 

77 



0 
0 

u 
11 



Marginal 
Total s 



Nominal Marginal 
Agreement Agreement 



♦ 

♦ 
♦ 

♦ 
♦ 



6 
71 

13 

90 



♦ 
♦ 

♦ 
♦ 
♦ 



• 33? 



• 333 

• 922 

• 8*6 



Llgnt»8 
Extent I on 
of Kappa 



• 318 



1,000 
.825 



9 

iRIC 



category Marginal Agreement! # f01 
Cohen's Kappel «782 



572 



573 



Table 296- 



Category chosen 
by Export 90 



Used 

Not used 
Other 

Mars I na 



Phate IV - XOE Evaluation Project, 
Descriptive Study - Spring 1978 

• Re I lab I II ty~on Materials 
Observer 20 Expert 90 
School 80i ; Grade 5 
Week 9 Day 5 
OMP' 

t 



0 



2 

Mat^r |a 



I j Other 



Cat 



% 



Category chosen 
by Observer 20 

U 2 3 : 



Marfllna* Nominal 
Total s Agreement 



1 ♦ * 0 6 0 

2 ♦ 0 ?X 0 

3 ♦ 0 1 12 
♦ 

♦ 

Totals ♦ 0 78 ,12 



♦ 



6 
71 
13 

90 



4 

> 



0,000 
1,000 
,923 



Maro I nal 
Agreement 



0,000 
• 910 
.923 



574 



/ 



category Marginal Agree^entl 
Cohen's Kappal .738 



611 



0c, 



Light's > 
EXtens | on 
0*4 Kappa 

r 

i 

0 ,000 
' 1,000 
, .911 



575 



9 

ERIC 



Table 297 * 



Phase IV «• IGE Evaluation Project 
Descriptive Study - spring 19,78 

Re I i ab I 1 1 ty on Mater I a I s 
Observer 10 Expert 90 , 
School 801 Grade, a 
Week 9 Day 5 
WDRSD 



Mater I a I I Other 



Category chosen 
by Expert 90 



Cat, 
* 



Cateoory chosen 
by Observer lo 



Mara I na 
Total s 



Nom | na I 
Agreement 

























Used 


1 




0 


1 


0 




1 ♦ 


0 


,000 


1,000 
























Not used 


2 


♦ 


1 


55 


1 


♦ 


57 ♦ 




,965 


,982 






















,889 


Other * 


3 


> 


0 


0 


8 


* 


8 ♦ 


1 


• 000 




















t 








+ + + + + + + + ■»•♦*♦♦ 


















♦ 








♦ 








! 


>arg 1 na 1 


Tota 1 s 




1 


56 


9 


♦ 


66 ♦ 












♦ 



















♦♦♦♦♦♦ 



L loht i s 
Marginal Extension 
Agreement of Kappa 



-,015 
,768 
1,000 



category Marginal Agreement! ,957 
CoKen's Kappal ,819 



577 



4 ' 



r 



Table 298 

Phase IV - XGE Evaluation Projedt 
Descriptive Study - Spring 1978 

Re I I ab I I 1 1 y on Mater I a I s 
Observer 20 Expert 90 
School 801 Grade 2 
Week 9 Day 5 
WORSD 



u 

Ui 

o 



Material! Other 



Category chosen 
by Expert 90 



Cateaory cho'sen 
by Observer 20 ^ 



Cat 
u 



78 



1 2 3 



Marolnal Nominal Marginal 
Totals Agreement Agreement 



Used 


1 




0 


1 


0 


♦ 


l 




0 


,000 


0.000 


Not used 


2 




0 


57 


0 




57 




1 


.00 0 


.983 


Other 


3 


♦ 


0 


0 


e 


♦ 


8 




1 


.000 


1.000 




































♦ 













Marginal Totals ♦ 



58 



8 



+ 
♦ 



66 



+ 



Category Marginal Agreement f*^. 661 
Cohen's Kappal .933 



L I ght » s 
Extens | on 
of Kappa 



0,000 
1,000 
1.000 



57 J 



/ 



4 >Table 299 



Phase IV - I GE Evaluation Project 
Descriptive Study - Spring 1978 

Re I I ab l I 1 1 y on Materials 
Observer 10 Expert 90 
School 801 Grade 2 \ 
Week 9 Day 5 
DMP ( 



Mater I a I I Other 



Category chosen 
by Expert 90 



Used 

Not used 
Other 

Marg I na I 



Cat , 
n 



Category chosen 
by Observer 10 



Mars ! na 
Tota I s 

















} 




1 


0 


0 




1 
















2 




1 


72 


3 




76 
















3 


♦ 


0 


3 






7 


















Nom | na I 
Agreement 



1,000 
,9^7 
,571 



Ligbt»s 
Marginal Extension 

Agreement o* Kappa 



1,000 
,509 

1,000 ,532 



,500 
*987 



category Marginal Agreement! ,829 
Cohen's Kapoal ,549 



5S1 



[Table 300 

Phase IV • IOE Evaluation Project 
Descriptive gtiudy - Spp I ns 1978 

w ' 

Re I I ab I I I t y on Mater I a I I - 
Observer 20 Expert 90 
School 801 .. Grade 2 , 
Week 9 Day 5 
DMP ' 

» 

Mater Jail Other 



Category chosen 
by Expert 90 



Used 

Not used 
Other 

Marg t na 



Cat , 
ft 



Cateaory chospn 
by Observer 20 



1 



Marginal Nominal 
Totals .Agreement 



L loHt » 8 

Marginal -—Extension 
Agreement of Kappa 



+ + + + + 4.+ 4. + + + + + + + + + + + ++ + + + + + + + + 

* ♦ ♦ 

1 ♦ 0.1 0 ♦ 1 ♦' 
+ ♦ ♦ 

2 + % 1 71 4 ♦ 76 ♦ 



4 

3 ♦ 
4 



4 

•4 



♦ 
♦ 



0,000 
,934 
.714 



1*000 
,974 
.778 



-.012 
, 447 
,680 



4 ♦ ♦ 

Totals ♦ 1 74 9 ♦ 84 ♦ 
♦ * * 



category Marginal Agrecmentl ,917 
Cohen's Kappal ,509 



5 b J 



ERIC 



# 



Table 301 

Phase IV - IGE Evaluation Project 
Descriptive Study - spring 1978 

* 

Re I I ab I I 1 1 y on Pater I a I s 
Observer 30 Expert So 
School 801, Grade 2 
Week 9 Day 5 

' DMp 



Mater I a I i Other 



s 



Category chosen 
by Expert 80 



< 



Used 

Not used 
Other 



Cateqofy chosen' 
by Observer 30 



Cat « 



Marginal Totals 



Marg I na I 



Nom | na 



Marg I na 



Llghtff 
Extens | on 



Totals Agreement Agreement -of Kappa 




0.000 
1 .000 
1.000 



t 



0.000 
,988 
1.000 



0 e 000 
1.000 
1,000 



9 

ERIC 



584 



category Marginal Agreement t , 663 
Cohen** Kappa! • 6£«2 



table 30 7 

Phase IV - iflE evaluation Project 
Descriptive Study - iSPMng 1978 

Reliability on, Mater I a I s * 
Observer 40 Expert 8o 
School 801 Grade 2 
Week 9 Day 5 
DMP 



Mater I a I l Others 



Category cKcs^n 
by Expert 80 



Used 

Not used 
Other 



5^ 



Cat , 
H 



Category chosen . s 
by Observer/ \0 

Maroin\a 



1 



• Tota I S\ 



v 



Norn | na I 
Agreement 



Marg | na I 
Agreement 



Marqinal Totals* 



1 



ao 



3 ♦ 84 ♦ 
♦ ♦ ♦ 



Light «s 
Extens | on 

of Kappa 



\+ ♦ + + + + + + + + + ♦ 4*+ ♦ + + ++* + + ** + t + * + 


■ * 








• * 


* 












1*1 0 


0 




1 




1,000 


• 1 .f 00 


1.000 


■ ♦ ^ ... 












** 




2 ♦ 0* 80 


0 




80 




1.000 


1,000 


1 ».000 


















3 + 0 0 


3 




3 




1.000 


1.000 


1 ,000 






♦ 













587 



category Marginal Agreementl 1.000 
Cohen's Kappal 1.000 



9 

ERIC 



Table 303 

Phase IV - !0C Evaluation Project 
DaterlPtlva study • spring 1976 

Reliability on learner Movaa 
Observer 10 Expert 90 
School 801 Orada 8 
waak 9 Oay 4 

DMP 



Category choien Cat t 
by exoart 90 n 



Cataoory choten by Observer lo 



UlOHt^l 

Harol nal Nominal Heroine! Extant Ion 
rotate Agreement Agreement o# KePPa 



♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦❖♦♦^♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦^ 
♦ 



Engaged » written 


1 




21 


1 


7 


0 


0 


0 


0 < 


. 29 ♦ 


,7£« 


,897 


,612 


Engaged • oral 




♦ 


0 


S 


J 


1 


1 


2 


0 * 


► 8 ♦ 


.375 


,750 


,330 


Engaged - eovart 


3 




5 


1 


5 


2 


0 


2 


0 < 


. IB ♦ 


.333 


,769 


,165 


Not engaged - Interim 


* 


♦ 


0 


0 


2 


IS 


0 


1 


0 i 


► 15 ♦ 


• BOO 


.862 


,753 


Not engaged • wa| t Ing 


3 




0 


i 


1 


0 


0 


1 


0 < 


► 3 ♦ 


o.ooo 


,333 


• •Oil 


Not engaged »oM teik 


6 


♦ 
♦ 


0 


0 


3 


1 


0 


11 


0 < 


. 18 ♦ 


.733 


,682 


,671 


Nona of the above 


7 


♦ 


0 


0 


0 


1 


0 


0 


4 < 


► S ♦ 


.BOO 


• BOO 


,T»1 
















Marginal Total! 


♦ 


26 


6 


19 


17 


1 


17 


4 < 


► 90 ♦ 









♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 



♦♦♦♦♦♦♦♦ 



Category Marginal Agraementl #762 
Cohtntg Kappal ,S39 



5 '.sb 



/ 



. Table 304 . 

rhasa V4 » iGt tvrluation Projact 
U6SC^Ih;!V'« r:jdy » SPf-no 1976 

Reliability on laarnar Move! 
Obtarvar 20 Expert 90 
School B01 Grade 9 
Week 9 Day % 
DMP 



Category e*otan 
by CxP«rt 90 

Cngagad » wr 1 1 tan 
Engagad • oral 
fcnaaged - covart 



Catt 



Category choaen by Obaarver 20 
1 2 3 4 5 6 



Marginal 
Total! 



Nominal 
Agraement 



Marginal 
Agraament 



UwSt'a 
Octant I on 
of KapPa 



1 ♦ 

2 ♦ 

3 ♦ 
♦ 



22 

0 



3 

0 
0 
1 
0 



0 

* 

3 

0 
0 

1 

0 



Not angagad • Interim 4 ♦ 

Not engaoad • waiting 5 ♦ 

♦ 

Not engaged -off teik 6 ♦ 
Nona of tha abova T ♦ 

Marginal Total! ♦ 

♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 



6 


0 


0 


1 


O 




89 ♦ 


,759 


,697 


,661 


0 


0 


2 


2 


O 


♦ 
♦ 


8 ♦ 


♦ 500 


1,000 


,451 


6 


1 


0 


0 


0 


♦ 


IS ♦ 


,533 


,662 


• 425 


1 


12 


0 


1 


1 


♦ 


15 ♦ 


# eoo 


,633 


,750 


I 


1 


1 


0 


0 


♦ 
♦ 


3 « 


,333 


1,000 


,310 


1 




0 


T 


1 




15 < 


,467 


,733 


,392 


0 


0 


0 


0 


3 




5 < 


► 1,000 


,714 


1,000 



26 



6 



17 16 



11 



♦ 

7 ♦ 90 



category Marginal Agreementi #666 
Cohtn't Kappal ,174 

58!) 



er|c 



I* 



Table 305 

Pha9a IV - IOC Evaluation Project 
^fraicrlPtlva study - spring 1976 

RtH«bli:tV on Leernar Movti 
Obtarvar Expart SO 

Sckool 60 T l Orada 3 
Waak 9 Day 4 

OMP 



Cataoory choton by Obtarvar 10 



Category eHoian Cata 
by Expart 80 * 







♦ 




Enoagad * wr 1 ttan 


1 


♦ 
♦ 


13 


Engaged <• oral 


2 


♦ 
♦ 


0 


Engaged • covert 


3 


4 
♦ 


1 


hof engaged - imarlm 


k 


♦ 
♦ 


0 


Not engaged • waiting 


5 


♦ 
♦ 


0 


Not engaged «oM talk 


6 


♦ 4 
♦ 


a 


Nona of tha abova 


T 


♦ 


0 


Marginal Total • 


♦ 
♦ 
▲ 


16 


• 




♦♦♦♦♦♦ 



0 
2 
1 
0 
0 

1 

0 



27 



Marg Inal 
total • 



Nominal 
Agreement 



MarO Inal 
Agraement 



1 


1 


0 


0 


0 


♦ 


15 « 


• 667 


• 936 


0 


0 


0 


0 


0 


♦ 


2 * 


liOOO 


• 500 




0 


4 


2 


0 


♦ 


30 « 


• 733 


• 900 


0 


2 


1 


0 


0 


♦ 
♦ 


a 4 


• 667 


• 750 


* 


0 


11 


0 


0 


♦ 
♦ 


15 « 


► *731 


• 662 


0 


0 


1 




0 


♦ 
♦ 


13 < 


► ,692 


• 646 


0 


1 


5 


0 


6 


♦ 
♦ 


12 < 


► .500 


• 500 



22 1 1 



♦ 
♦ 



90 



Extantlon 



,B36" 
1 • OOP 
,619 
,651 
9 6*T 
,6*9 
^ .666 



category Marginal Aoraamtfiti ,731 
CohaMf Kappai ,693 



FRir 



5<J() 



I 



Table 306 

Phase IV • IOE Evaluation Project 
Descriptive Study • SPrjnQ 1978 

Reliability on learner Moves % 
Observer 10 Expert 90 
school 601 Grade I 
Week 9 Day 4 

WDP8D<* 



Category ehosen 
by expert 90 



Catt 



Category chosen by Observer 10 

1 2 3 * V 6 ' 



Marginal 
Totan 

♦♦♦♦♦♦♦♦ 



Enoegad • wr I tten 


1 


♦ 
♦ 


3 


0 


1 ' 


0 


0 


0 


0 < 


4 ' ♦ 


(750 


1.000 


Engaged * ^ffll 


2 




0 


3 


2 


1 


0 


0 


0 « 


8 ♦ 


• 500 


.667 


Engaged • covert 


3 




1 


1 


32 


0 


2 


2 


0 < 


> 35 < 


.5*2 


,97* 


sot engaged - interim 


4 




0 


0 


0 


0 


0 


0 


0 < 


► 0 < 




0.000 


Not engaged « waiting 


5 


♦ 


0 


0 


0 


0 


0 


0 


0 < 


► 0 4 




0.000 


Not engaged »oM task 4 


6 




0 


0 


e 


1 


0 


3 


0 < 


► 5 < 


> .600 


.533 


Nona o# the above 


7 




0 


0 


2 


1 


1 


0 


8 - 


> ie < 


► .6*7 


,»f,7 



♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦ ♦♦♦♦♦♦♦♦♦♦♦♦ 



Marginal Total* ♦ 
♦ 



39 



66 

♦♦♦♦♦♦♦♦ 



LIOHtU 

Nominal Marginal Eicttntlon 
Agreement Agreement o# KaPPa 



,734 
,61^ 



,469 
• 681 



Category Marginal Agreement! #892 
Cohen* • Kappei ^ ,668 

O w 1 



ERIC 



0 



Table 307 k 

Phase IV - loi Evaluation Project 
Descriptive Study - Soring 1976 

pel lability on Learner Moves 
Observer 20 Expert 90 
School 601 Orede I 
Week 9 Day 4 

WDRSD 



CotOQory chosen 
by rxPert 90 



Cat, 
U 



Category chosen by Observer 20 
1 8 S . * 5 6 7 
♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 



Engaged - written 


1 


♦ 
♦ 
♦ 


3 


0 


0 


0 


0 


0 


1 


Engaged • oral 


8 


♦ 
♦ 


0 


4 


1 


1 


0 


0 


0 


Engaged • covert 


3 


♦ 
♦ 


0 


a 


3* 


0 


1 


1 


0 


Not engaged - Interim 


* 


♦ 
♦ 


0 


0 


0 


0 


0 


0 


0 


Not engaged - waiting 


5 


♦ 


0 


0 


0 


0 


0 


0 


0 


Not engaged -of* task 


6 




0 


0 


3 


0 


0 


3 


0 


None of the above 


7 


♦ 


0 


0 


0 


0 


0 


3 


9 


















Mr-g|nal Totals 


♦ 


3 


6 


36 


1 


1 


7 


10 



Merglnel 
Toteis 

♦♦♦♦♦♦♦♦ 

6 
36 

0 

0 

6 
12 

♦ 

66 

♦♦♦♦♦♦♦♦ 



Nomine | 
Agreement 



#750 
#667 
#695 



#600 
#750 



Marginol 



Uoht'e 
Extension 



Agreement of KepP« 



,750 
1 # 000 
1#000 
0t000 
0*000 
#667 
#633 



t 73B 
#633 
t 752 



# 7Q* 



Category Marginal Agreement! #634 
Cohems Keppei ,668 



9 

ERIC 



592 



■ Table 308 

PheM IV - IOE Evaluation Projact 
DaicMPtlve Study * Spring 1970 

Reliability on t«arnar Movat 
Obsarvar 40 Expart 60 
.School 601 Orada 3 
utak 9 Oav 4 

WDRSO 



Catago'V chotan 
by Expert 60 



Engaged • wrlttan 
Engagad - oral 
Enoaaed - covart 



cataoory choian 3 by Obtarvar *0 



Morgl nal 
Totals 




Nominal Marginal 
Agraamant Agraamant 



ugHt'i 
Extant Ion 
of KaPPa 



1 ♦ 
♦ 

3 ♦ 
♦ 

3 ♦ 
♦ 



fcot angagad • Mtarlw * ♦ 



6 

\ 
1 
o 

0 
0 
0 



0 
3 
0 
0 
0 
0 
0 



Not angagad - waiting 6 ♦ 

♦ 

Not angagad «o#f ta$k 6 ♦ 

♦ 

Nona o# tha abova T ♦ 

♦ 

Marginal Total* ♦ 10 3 
♦ 



3 

1 

13 
0 
1 

0 

0 




♦♦♦♦♦♦♦♦♦♦♦♦ 



0 


0 


0 


0 


♦ 
♦ 


10 ♦ 


• 600 


1,000 


,7*0 


0 


0 


0 


0 


♦ 
♦ 


5 ♦ 


J*' 

• 600 


,600 


,5T9' 


3 


3 


0 


0 


♦ 
♦ 


IT ♦ 


• 706 


,9*1 


,899 


3 


0 


0 


0 


♦ 
♦ 


S « 


ItfOO 


,600 


1,000 


0 


6 


0 


0 


♦ 


7 < 


► iHT 


,«75 


> ,039 


0 


0 


0 


0 


♦ 


0 i 






\ 


0 


0 


0 


16 


4 

♦ 


16 


► 1,000 


1,000 


4,000 



catagory Marginal Agraamantl «636 
Cohan«i Kappal «787 



Table 309 



phase IV - IGE Evaluation Project 
DoicrlPtlv* Study - Soring 1978 

Riil lability on Learner Movet 
Obierver f£ Expert 90 
School 001 Grade 8 
Week 9 Day 4 

DMP 



Cataooey choten by QbMrver 40 

5 6 7 



Category choien Cat* 
by expert 90 * 




1 


2 


3 


4 


Engegod • written 


1 


♦ 
♦ 
♦ 


12 


0 


12 


0 


Engaged • oral 


2 


♦ 
♦ 


1 


1 


3 


0 


Engaged - cover v t 


3 


♦ 
♦ 


9 


a 


10 


0 


Not angagad « Interim" 


4 


♦ 
♦ 


0 


0 


0 


3 


Not angagad » waiting 


5 


♦ 


1 


0 


1 


0 


Not engaged *o*# taiM 


6 


♦ 

♦ 


0 


1 


1 


0 


None of the above 


7 




0 


0 


0 


0 



♦ 

Marginal Toteia ♦ 19 4 



i 
o 
i 

0 
5 
4 
0 



0 
0 
2 
0 
1 
9 
0 



0 
0 

1 

0 
0 
0 
7 



Marginal 
Total • 



27 



11 12 



♦ 

♦ 

♦ 
♦ 

♦ 

♦ 
♦ 
♦ 
♦ 
♦ 

♦ 

♦ 
♦ 
♦ 



25 
5 
21 

3 

e 

is 

T 
6* 



Nominal Marginal Extan»lon 
Agraamant Aox««mant o* K«PPa 



• 480 


*760 


• 326 


• 200 


• 800 

t 


• 160 


• 476 


• 778 


,226 


1*000 


1 *000 


1.000 


.625 


• 727 


♦ 568 


• 609 


• •00 


• 533 


ItOOO 


• 875 


l t 000 



Category Marginal Agreement I .620 
Cohente Koppei .451 



594 



9 

ERIC 



Table 310 

Phase IV - IOE Evaluation Project 
OatcMPtlve study - spring 1978 

Reliability on Learner Moves 
Observer 20 Expert 90 
school aoi Orada 2 
tteek S Day * 
OMP 



Category chosen Cat# 
by expert 90 * 



tngagnd • Vrltten 
Engaged • ore If 
Engaged • covi 



Category chosen by Observer 20 
12 3*567 



Mar g I ne I 
Total i 



Nominal Marginal 



Lights 
Extension 



♦ I 

1 ♦ 19 
♦ 

2 ♦ 0 
♦ 

/*3 ♦ 5 
♦ 

Not engaged • 'Inform 4 0 

Not 'engaged • wafting 5 ♦ 1 

♦ 

Not engaged -off task 6 ♦ o 

♦ 

Nona of the ebove T ♦ 0 

♦ 



Marginal Totals ♦ 25 
♦ 



i 

\ 
o 

0 

1 

0 



3 

0 

u 

0 

1 

2 

0 

17 



0 
0 
0 
3 
1 
0 
0 



0 
0 

1 

0 
5 
3 
0 



2 
1 
2 
0 
0 
9 
0 



14 

f 



0 
0 

1 

0 
0 
0 

y 



25 

5 
21 

3 

e 

15 
7 

♦♦♦♦♦♦♦♦ 
6* 

♦♦♦♦♦♦♦♦ 



tartcmant Aor«*m«nt of k«pp« 





/) 

/ 




,760 


1.000^ 


,656 


,600 


.71* 


,768 


,584 


,610 


,403 


l.OOO^f 


,750 


rtooo 


,625 


.609 


.560 


,600 


,933 


,5S0 


i\ooo 


,6TB 


1,000 




Category Marginal Aoraementi #853 
Cohen*i Kappal #616 



Table 311 



PhaM IV - XOE Evaluation Project 
DetcMPtlve tfftdy - soring 197B 

Reliability on L«orner Movaa 
observer UQ CKPart 80 
school eo: orade a 
week 9 Day * 

OMP 



Catagory chotan 
by EXPart 80 



Engaged - wr I t tan 
Engaged - oral 
Engaged • covart 
Not engaged - Interim * 
foot engaged -waiting 5 
hot engaged -off talk 6 



N^ne o# the above 



Category choitn by ObMrver 10 

1 



marginal Total) 




Lights 

Marginal Nominal Marginal Jjtjjjjj" 
Totait Agreament Agraement of Kappe 



13 ♦ 


,615 


1,000 


,536 


4 ♦ 


,750 


,333 


,705 


38 ♦ 




,632 


,2*0 


6 ♦ 


,500 


,633 


^66 


7 » 




,700 


/,3*i 


6 ♦ 


,,633 


,500 


# 603 


6 « 


,850 


,500 


,230 



category Maralnal A«r««mtntl ,6*3, 
COh«n»t K«PP«1 ,3*6 



9 



596 



Table 312 

Phata IV *• IOE evaluation Projact 
OaicrlPtlva Study * spring 1978 



, Ral labl I I ty on learner 
Observer 10 Expert 
School tfoi Orada 
Haak 9 Day 5 

DMP 



Move* 

OA 



Category phot.tn 
by Export 90* 



Engaged - written 
Engaged • oral 
engaged - covart 



hot engaged -of t tetk 
Nona of the abova 



Cat r 
# 



Catagory chosen by Observer l<p 

j ' 2 3 a 9 6 T 



Marginal 
Total i 

♦♦♦♦♦♦♦♦ 



♦ 

1 ♦ 


14 


0 


4 


0 


1 


0 


0 « 


19 ♦ 


j tTST 


,947 




,674 


♦ 

2 ♦ 


0 


0 


0 


0 


1 

0 


1 


0 « 


14 


0,000 


1,0 0 0 




• ,oU 


♦ 

3 ♦ 


1 


0 


14 


I 




6 


0 < 


> 22 < 


,636 


,955 




,526 


♦ 

4 ♦ 


1 


1 


0 


6 


0 


0 


0 « 


> 6 < 


► ,750 


1,000 




• T26 


♦ 
















► 0 < 




0,000 






5 ♦ 


0 


0 


0 


0 


0 


0 


0 < 










,667 


♦ 

6 ♦ 


2 


0 


3 


1 


0 


21 


0 


» 2T < 


► i7T6 


,900 




♦ 

T ♦ 


0 


0 


0 


0 


0 


2 


11 


♦ 13 


» ,646 


,046 




,621. 




Lights 

Nominal Marginal Eaten* Ion 
Agreement Agreement ^of^KaPPa 



♦♦♦♦♦♦♦♦ 
90 

♦♦♦♦♦♦♦♦ 



Catagory Marginal Agreementl .607 

Cohema Keppei #656 5137 



Table 313 

Phasa IV • IOC f valuation Projtct 
OticclPtlva Study - spring 197B 



Rtl | obi I I fcV on 
Obtarvar 20 

school eoi 

Waak ? 



Laarnar Movoi 

Expart 90 
*Grada 8 
Day 5 

OHP 



Category cho«n 
by Expart 98 



Ehgagad • wrlttan 
tnoagtd - oral 
Cnoagad «• covart f 



Cett 



cataoory <5h*M*sby Obiorvar 80 



Marginal 
ToUlt 



r 

♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦^♦^ 



1 ♦ 
3 ♦ 



J*6t anoagad • Intarlm^ * ♦ 

Ivot ang»g«d • waiting 5 ♦ 

Hot angagad *oM- tttta 6 ♦ 

Nona of tha abdva 7 ♦ 

« ♦ 

\ 



V 



is 



0 
jr 0 



i 

0 

6 

1 

0 

2 

0 



0 

\o 
t 

6 
0 
0 



80 



\i 12 



♦ 
♦ 

♦ 
♦ 

♦ 

♦ 
♦ 
♦ 



19 
1 

22 
8 

0 
27 
13 



Nominal Marginal 
Agrggwant Agrggwgnt 



26 12 ♦ »° 

♦♦♦♦♦♦♦♦♦♦♦♦♦ 



Category Marginal Agraamantl #T8^ 
Cohgntt KaPPil < **3l 




V 



,9*7 
,000 
,364 
,780 

.741 
,983 



,633 
1,000 

,545 
1,000 
0,000 

,963 

,923 



LlqhtU 
Extension 
of KaPP«' 



.921 

4 

.,011 
,266 
.72* 



,*3« 



9 

ERIC 



598 



Table 314 

phate IV - IOE Evaluation Project 
0«tcrlPtlve Study - spring 1*T8 

r«| lability on.Loerner Moves 
obiervar 30 Expert SO 
School 801 Grade S v 

DMP 1 



Category chosen by Observer 30 



Category choee* 
by e*P* r * 80 



tngeoad^in^ tten 

engaged p oral 

Engaged - covert 

hot engaged • Interl^ 

Not engaged • waiting 5 ♦ 

♦ 

Not engaged -oM task 6 ♦ 
None of th* above 




1 ♦ 
♦ 

2 ♦ 
♦ 

3 ♦ 
♦ 

* ♦ 



7 ♦ 

♦♦♦♦♦♦♦♦♦♦♦ 
♦ 

waromal Totait ♦ 28 T 



20 


I 


1 


0 


0 


0 


0 


1 


5 


0 


0 


0 


T 


2 


18 


0 


3 


1 


0 


0 


0 


2 


0 


0 


• 


0 


0 


1 


12 


3 


0 


3 


1 


0 


1 


5 


0 


0 


0 


1 


1 


0 



Marginal Nominal werg|ne| 
Total! Agreement Agreement 



♦ 
♦ 
♦ 
♦ 
♦ 
♦ 
♦ 
♦ 
♦ 
♦ 
♦ 

♦ 
♦ 



ught»t 
extension 
of KepPe 




22 ♦ 


• 90« 


,786 


,668 


6 ♦ 


,167 


,887 


,096 

4 


31 * 


• 601 


,806 


,*19 


2 * 


1,000 


,500 


1,000 


17 * 


► . ,706 


1,000 


,637 


10 < 


► N ,500 


,900 




2 


» 0,000 


0,000 


8,000 - 



CaUaory Mareinal Aaraamantt .693 
Coh«n»« Kappal #8*1 



Table 315 

Phaia XV - !0E Evaluation project 
OaacHPtlve Study - Spring 1976 

Reliability on Laarnar Movaa r 
Obaarver 40 , Expart 80 
school * Grada 5 

w*%k 9 Day * 

DMP 



Cfrtajory cHotan 
by Expart bo 



Enoagad • written 1 ♦ 

Engaged • oral 2 ♦ 

♦ 

Engaged «■ eovart 3 ♦ 

Not angagad • intarim 4 ♦ 

♦ 

Not angagad -waiting 5 ♦ 

♦ 

♦Not angagad -off tai* 6 ♦ 
Nona of tha abova T ♦ 



Catagory choaan by Obaarvar 40 
I 2 3 * 5 6 



Marginal Totalt ♦ 23 6 
♦ 

♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 



i 

< 



Marginal 
Totalt 




LlQht't 

Nominal Marginal ' E»«tanilor> 
Agraamant agraamant o? KOPP» 




0 


♦ 
♦ 


22 ♦ 


• 773 


• 957 


• 695' 


0 


♦ 
♦ 


6 ♦ 


• 333 


1,000 


,26* 


0 


♦ 
♦ 


31 ♦ 


• 742 


• 838 


•562 


0 


♦ 


2 ♦ 


1 • 0 0*0 


9 500 


\ la 000 


0 


♦ 


IT « 


• 586 


• 568 


,537 


0 


♦ 
♦ 


10 « 


,600 


• 800 


• 561 

* 


z 


♦ 
♦ 


2 i 


► 1*000 


uooo 


1.000 
















♦ 










2 


♦ 


90 









Category Marginal agraamanti' .018 
Cohan»i Kappal .900 



9 

ERLC 



6U0 



( 



Table 316 

Phase IV - I GE Evaluation Project 
Descriptive Study - Spring 1978 

Reliability on learner novas 
Observer 10 Expert 90 
school eoi orade 2 
Week 9 Day 5 

W0R5D 



Category chosen Cat* 
by E*P«rt 9 0 v * 



Category chosen by Obnerver 10 



6 



Marginal 
Totals 



Nominal Marginal 



UgKt's 
Extens Ion 



Agreement Agreement of KepPe 



Engaged - wr 1 t ten 


I 


♦ 
♦ 


s 


2 


l 


0 


0 


1 


0 


♦ 94 


« 556 


• 8)8 


,467 


Engaged • oral 


2 


♦ 
♦ 


0 


1 




0 


■ 0 


0 


0 


♦ *3 ♦ 


• 333 


• 375 


• 8*1 


Engaged - covert 


3 


♦ 
♦ 


2 


k 


10 


1 


0 


0 


0 


♦ 25 ♦ 


• 720 


1,000 


,5*9 


Not engaged - interim 


4 


♦ 


1 


0 




0 


1 


G 


2 


♦ 6 ♦ 


o^ooo 


• 167 


• •01* 


Not engeged « waiting 


5 


♦ 

♦ 


2 


0 


1 


0 


1 


1 


0 


♦ 5 ♦ 


• 200 


• 600 


• 162 


Not engeged *otf task 


6 


♦ 


1 


1 


1 


0 


1 


6 


0 


♦ 10 ♦ 


#600 


• 800 


•**s 


None o# the above 


T 


♦ 
♦ 


0 


0 


0 


0 


0 


0 


a 


♦ 8 ♦ 


iiOoo 


• 800 


1*000 






♦ 
























Marginal Totals 


♦ 
♦ 


11 


e 


25 


1 


3 


e 


10 


♦ e* ♦ 









♦♦♦♦♦♦♦♦ 



Category Marginal Agreement I •ftBl 

Cohen's Keppei ,480 GOl 



9 

ERLC 



Table 317 



phasa IV - !QE Evaluation Projact 
DatcrlPtlva Study - Spring 1978 

Ral lability on Laarnor Movat 
Obtarvar 20 Expart 90 
School 601 Qrada 2 
waek 9 Day S 
WDRSD 



Catagory chctan by Obtarvar 20 



Catagory choian 
by E*P8r* 90 



Cat • 



♦♦♦♦♦♦ 







♦ 








Ensegad « wrl ttan 


1 


♦ 


6 


0 


0 


Cnoagad • oral 


2 


♦ 

♦ 


0 


2 


1 


Engagad • covart 


3 


♦ 
♦ 


4 


4 


16 


Not angagad • |ntar|m 


4 


♦ 
♦ 


1 


0 


0 


Not angagad • waiting 


5 


♦ 
♦ 


1 


C 


1 


Not angagad »oH task 


6 


♦ 


0 


0 


1 


Nona o# tha abova 


T 


♦ 
♦ 


0 


0 


0 


Marginal TotaH 


♦ 


1* 


6 


19 



0 
0 

1 

3 
1 
0 
0 



s 



1 

0 
0 
2 
1 
3 
0 



0 
0 
0 
0 

1 

6 

0 



0 
0 
0 
0 
0 
0 



Marginal 
Total s 




♦ 

♦ 
♦ 
♦ 

♦ 
♦ 
♦ 
♦ 
♦ 
♦ 
♦ 



66 



UoHt'a 

Nominal Marginal 
Agraamant Agraamant o* K«PP« 



9 ♦ 


«e©9 


• 643 


• 869 


3 ♦ 


9 667 


• 900 


• 63^ 


25 ♦ 


9 640 


• 760 




6 ♦ 


.500 


• 833 


• 459 


5 ♦ 


.200 


.71^ 


• 108 


10 « 


• 600 


• 700 


• 553 


8 < 


► I. 000 


1.000 


1,000 



C«t««ory Marginal *or««m«f>ti .m 



9 

ERIC 



coy 



/ 



Table 318 

phaie IV • I GE Evaluation Project 
DetcrlPtlve Study - Spring 1976 

Reliability on Learner Movet 
Observer 30 Expart BO 
School 601 Grade a 
Week 9 Day 5 

WDRSD 



Category ehoten Cat. 
by Expert 60 * 



Category chosen' by Observer 30 



1 



Lights 

Merglnel Nominal Marginal EKtentlon , 
Total! Agreement Agreement o# Keppa 



Engaged - wr 1 tten 


I • 


► 13 


0 


0 


0 


1 


0 


0 


♦ 
♦ 


I* 


Engaged » oral 


2 i 


► 0 


1 


0 


0 


1 


0 


0 




2 


Engaged ~ covert 


3 


¥ 5 


1 


10 


0 


6 


1 


0 




23 


Not engaged • Inter Im 




♦ 0 


0 


0 


0 


0 


0 


0 


♦ 


0 


Not engaged • wel t 1 ng 


5 


♦ 0 


0 


0 


0 


T 


1 


0 


♦ 


6 


Not engaged -off task 


6 


♦ 1 


0 


2 


0 


3 


3 


0 




9 


hone of the above 


T 


♦ 0 


1 


0 


1 


0 


0 


t 


♦ 




















♦ 


► ♦■u 


Marginal Totals 


♦ IV 


3 


12 


1 


16 


5 


2 


♦ 
♦ 


§0 



#929 


• 737 


,695 


• 500 


• 667 


• *76 


• 435 


• 522 


• ■•» 


• 675 


0 # 000 

•♦♦♦ 


,681 


9 333 


• 666 


• 273 


«600 


• 600 


.463 



Category Marginal Agreement) 
COhemt K6PP6) •*«* 



i*6* 

$02> 



Table 319 

Phate IV - IOC Evaluation Projact 
OaicrlPtlva study - Spring 1978 

flallablllty on Le»rn«r Movtn 
Obiarvar 40 Exoart 00 
School 801 , Oredt 2 
Waek 9 Day 9 

W0RS0 



Catagory cHoian 
by Ewpart BO 



Cat f 



Catagory chotan by Obtarvar 40 
18 3 4 5 6 7 



Marginal 
Total i 



U|«Kt«t 

Nominal Marginal Extant I Of 
Agpaamant Agraamant o* KapPa 



Engagad • wrlttan 
Engagad ■ oral 
Engagad - covart 



♦ 

1 ♦ 13 0 1 0 0 0 0 
♦ 

£♦1000100 

3 4 0 2 21 0 0 0 0 
♦ 

Not angaoad • intaMm 440000000 

Not angagad -waiting 5 ♦ 0 0 1 0 6 1 0 

♦ 

Not angtgad -o#t task 6 ♦ 2 0 3 1 2 I 0 

♦ 

Nona of tha abova 740002002 



Marginal Total" ♦ 1* 2 26 



♦ 
♦ 
♦ 
♦ 
♦ 
♦ 
♦ 
♦ 
♦ 
♦ 
♦ 
♦ 
♦ 
♦ 
♦ 

♦ 
♦ 



1* 
2 

23 
0 
6 
9 
4 

60 



#989 
0*000 

• 913 

• 750 

• HI 

• 800 



• 875 
ItOOO 

• 885 
0t000 

• 889 

• 222 

• 800 



• 903 
(# 034 

9 847 

• 706 

• 090 
•Ml 



Category Marginal Aoraamantl $624 
Cohantt Kappai #619 



GO 4 



Table 320 

Phase IV - IOE Evaluation project 
OescrlPtlve Study • soring 1978 

Reliability on L*amer Moves 
Observer 30 Expert SO 
School 601 orade 2 
Week 9 Day 5 
DMP 



Category chosen by Observer 30 



Category chosen Cat f 
bv Expert 80 * 



Margl rial 
Totan 



Light's 

Nomina) Marginal Extension 
Agreement Agreement' o# K«PPa 



Engaged - wr I t ten 
Engaged • oral 
Engaged - covert 
Not engaged • Interim * 
hot engaged • waiting 5 
hot engaged -oil task 6 
None of the ebove T 

Marginal Totals 



♦ 

♦ 
♦ 
♦ 
♦ 

♦ 
♦ 



13 

0 
* 

0 
0 
0 
0 

IT 



0 
0 
2 
0 
0 
0 
0 



2 
2 
25 
3 
3 
k 
0 

39 



0 
0 
0 

* 

0 
0 
0 



1 

0 
2 
1 
1 
1 
0 



0 

0 

1 
1 

0 

11 

0 
13 



0 
0 
0 
0 
0 
0 
3 



♦ 
♦ 

♦ 
♦ 
♦ 
♦ 

♦ 

♦ 

♦ 
♦ 

♦ 
♦ 



16 
2 

34 
9 
* 

16 
3 

$4 



> + ♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 



.613 


.941 


,765 


0.000 


1.000 


.,02^ 


.735 


,072 


,506 






,*1T 


,250 


,667 


,1« 


♦ 688 


,613 


,630 


1,000 


1,000 


1,000 ' 



category Margfnel Agreement! 9 820 
Cohen! a Kappa l *842 



9 

ERIC 



Table 321 

l>hsi« IV * IOE Evaluation Projoet 
DaiCMPtlva Study - spring 1978 

Rt! lability on Laarnar Movti 
Obitrvar 40 ExvrY 80 
School 801 Qr*49 9 
Waak 9 Oay S 

DMP 



Category cHoian 
by export 80 



Cat . 
* 



Catagory choian by Obiarvar #0 
I 2 3 4 S 6 



Marginal Nominal Marginal exttn^'on 
Tottli Aoroomant Agraamant of KaPPa 



Engaged • *r 1 ttan 


1 


♦ 

♦ 14 


0 


1 


0 


1 


0 


0 


♦ 
♦ 

♦ 


18 ♦ 


• 878 


• 898 


• 028 


tngagad • oral 


2 


♦ 

* 0 


0 


2 


0 


0 


0 


0 


♦ 


a « 


o.ooo 


• 500 


.,012 


Cngagad - covart 


3 


♦ 

♦ 9 


0 


21 


0 


1 


3 


0 


♦ 
♦ 




• 818 


• 653 




Not angaoad • mtarim 


4 


♦ 

4 0 


1 


1 


7 


0 


0 


0 


♦ 


9 i 


i778 


• 778 


,T5« 


Not angagad ■» wa 1 ting 


5 


♦ 0 


0 


3 


0 


0 


1 


0 


♦ 
♦ 




► 0i000 


• 750 


• »03T 


Not anoagad -o#f task 


6 


♦ 0 


0 


\ 


0 


1 


14 


0 


♦ 


18 < 


► .878 


• 889 


,0*1 


Nona o* tho abova 


7 


♦ 

♦ 0 
♦ 


0 


0 


0 


0 


0 


3 


♦ 


3 i 


► liOOO 


1,000 


1,000 






♦♦♦♦♦♦♦♦ 














♦ 










Marginal Totaii 


♦ 

♦ 23 
♦ 


i 


29 


7 


3 


18 


3 




84 









Category Marolnal Agraamanti ,TS1 
cohanti Kappa I .40* 



9 

ERIC 



GO (J 



Table 322 

Phata IV * roe Evaluation Projact 
De*criPtiva Study -^Spring 197B 

pal lability on Ltarnar Movat 
Obnarvar 10 Expert 90 
School 601 Orada i 
waak 9 Day 5 



Catagory ehotan Cati 
by export 90 * 



Catagory chotan by Obtarvar 10 



Marginal 
Totoit 



Nominal 
Agraawant 



Marginal 
AGraamant 



ItgHt'a 
Extantlon 
of Kappa 



Engagad • writ tan 


1 


4 


9 


0 


2 


0 


1 


I * 


0 


4 
4 
4 


13 ♦ 


♦ 692 


♦ 769 


.691 


Engagad • oral 


2 


♦ 


0 


1 


0 


0 


0 


0 


1 


« 4 
4 


2 ♦ 


» BOO 


♦ 667 , 


,461 


Engagad « covart 


3 


4 
♦ 


0 


1 


17 


3 


0 


2 


1 


>♦ 

4 


24 ♦ 


.708 


' ,800 


,846 


Not angaoad • IntarS* 




4 
4 


1 


0 


2 

r 


2 


0 


0 


1 


4 
4 


6 ♦ 


,333 


♦ est 


.273 


Not angagad « waiting 


S 


4 

4 . 


0 


1 


8 


1 


11 


4 


0 


4 


25 ♦ 


»440 


.920 


,337 


Not %f\QiBQ^4 «o## talk 


6 


4 

4 


- 0 


0 


1 


0 


1 


s 


0 


4 
4 


7 ♦ 


♦ T14 


.900 


,667 


Nona of tha abova 


7 


4 
4 


0 


0 


0 


1 


0 


2 


4 


4 
4 
4 


7 ♦ 


.■71 


1.000 


. .931 






♦44444444< 


• ♦♦♦♦44444444444444 4*444444444444444444 44 

A 4 








Marginal Totals 


4 
4 


10 


3 


30 


7 


13 


14 


7 


♦ 


84 ♦ 












4 














) 







catagory Marginal Agraamantl #730 
Cohan's Kappai »*t3 

R n 7 



Table 323 

Phaia IV - IOE rvaJuatlon Projoct 
ucicrlPt! *? Study • SPrino 1978 

Reliability on learner Movee 
Obtarvar 20 Expert 90 
School 601 Orada 2 
tfaak 9 Day 3 

DMP 



Cetegory ehoten by Obtarvar 20 



Category cHo«en 
by Expert 90 



Engaged • written 
Engaged • oral 
Engaged - covert 



Cat, 
a 



1 



♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 
♦ 



1 ♦ 

2 ♦ 
♦ 

3 ♦ 



6 
0 
1 



Not angagad - Interim * ♦ 

♦ 

Not angagtd - waiting S ♦ 

♦ 

Not angagad •oil talk 6 ♦ 

♦ 

Nona of the abova 7 ♦ 

♦ 

Marginal Total! ♦ 7 6 



4 

0 

1 
6 
1 
0 

27 



5 6 7 
2 0 1 



0 
0 
0 
16 

1 

0 

19 



0 

z 

0 
2 
5 
2 

U 



Marginal 
Totali 

13 
2 

* 

as 

7 

7 

♦ ♦♦<►♦♦♦♦ 
04 



Nominal 
Agreement 



,462 
,500 
,629 
,667 
,640 

• 714 

• 714 



Light't 
Marginal Extension 
Agreement of Kappa 



i538 
i333 
,669 
,633 
,760 
• 636 
,776 



,413 
9 462 
,447 
,646 
,535 
,671 
,660 



Cataoory Marginal Agreement! «iil 
Q0H«nvs Kappet t5i9 



608 



Table 324 



Phftift IV - IOE Evaluation project 
Datcrlptlva Study - spring 1978 

Reliability on intafact I ont 
Obtervar 10 Export 90 
School 801 Orada 5 
Week 9 Day * 

OMP 



Datet 06-85-79 
ProgramI RELP96 
Data r | III RELD1 



Category choien Cat» 
by ExP«rt 90 * 



Cataoory chotan by Obtarver 10 



8 9 



10 11 



Marginal Total! 



Merg I na | 
Totals 



t|ght«t 

Nominal Marginal Cxtanaion 
Agreement Agreement of Kappa 



Targat to Teacher 


1 ♦ 


1 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 « 


1 ♦ 


liOOO 


1,000 


1,000 


Target to Other Adul t 
Target to student 


2 ♦ 

3 ♦ 


0 
0 


0 
0 


0 

s 


0 
0 


0 
0 


0 
0 


0 

0 


0 


0 
0 


0 
0 


0 4 

s 4 


0 ♦ 
7 ♦ 


• 286 


• ST1 




Target to Tchr fc Orp 


A ♦ 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 4 


o ♦ 








Taacher to Target 


5 ♦ 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 4 


0 ♦ 








Other Adul t to Target 
Student to Target 


6 ♦ 

7 ♦ 


0 
0 


0 
0 


0 
0 


0 
0 


0 
0 


0 

0 


0 

1 


0 
0 


0 
0 


0 
0 


0 4 
2 4 


0 ♦ 
3 ♦ 


,333 


• 333 
0*000 


• 926 


Teacher to oroup 


8 ♦ 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 4 


0 ♦ 








Other Adul t to oroup 


9 ♦ 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 < 


► 0 v ♦ 








Student to oroup 

/ 

Nona of tM above 


10 ♦ 

11 ♦ 


0 
0 


0 
0 


0 

a 


0 
0 


0 
0 


0 
0 


0 
0 


0 

1 


0 
0 


0 
0 


0 i 
76 < 


► 0 ♦ 

► 79 ♦ 


<96ft 


,952 


• •12 



1 1 0 0 83 



90 

♦♦♦♦♦♦♦♦ 



GO'.) 



Catagory Marginal Aqrtamantl .871 
Cohan Kappat ,*9* 



610 



t 



Table 325 

Pbata IV - IOE Evaluation Projact 
DatcrlPtlva study - Spring 1978 

Ral labl I I ty on interaction* 
. obaarvar 20 Expart 90 
School aoi orad* 5 
Waok * Day <» 
DMP 



D«t«l 06-2S»79 
aroaroml Rf.LP95 
Oata F| lal RElOl 

; 



Cataoory choaar, by Ob«arvar 80 



Catagory e*o«an 
by ExPart 90 



Cat* 
a 















a* 


Target to Taachar 


1 


♦ 
♦ 


l 


Taroet to OtHar t 


Z 


♦ 
♦ 


0 


Targat to Studant 


3 


♦ 
♦ 


0 


Targat to Tchr L Orp 


* 


♦ 
♦ 


0 


Taachar to Targat 


5 




0 


Othtp Adult to Targat 


6 


♦ 
♦ 


0 


Studant to Targat 


7 


♦ 


0 


Taacsar to oroup 


0 


♦ 
♦ 


0 


Othap Adult to orou^ 


9 


♦ 
♦ 


0 


Studant to oroup 


10 


♦ 


0 


Nona of tha abova 


11 


♦ 
* 


1 






♦♦♦♦♦♦♦♦♦ 

a. 


Marginal Totan 


♦ 


s 



0 

0 

0 

0 

0 
0 
0 
0 
0 
0 
0 



0 

0 

3 

0 

0 

0 

3 

0 

0 

0 

i 



0 

0 

0 

0 

0 

0 

0 
0 
0 
0 
0 



0 

0 

0 

0 

0 

0 

0 
0 
0 
0 
0 



0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 



0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 



0 

0 

0 

0 
0 
0 
0 
0 
0 
0 

1 



0 

0 

0 

0 

0 

0 

0 
0 
0 
0 
0 



10 u 



0 
0 
0 
0 
0 

c 

0 
0 
0 
0 
0 



0 
0 

* 

0 
0 
0 

1 

0 
0 
0 

ra 
•1 



Marginal 
Totals 

1 

0 
7 
0 
0 
0 
3 
0 
0 
0 
79 

90 



Ughttt 

Nominal Marginal Extamlon 
Aor aamant - AOraamant of Kappa 



liOOO 



• 439 



• 800 



,657 



0*000 



0,000 



1 1 000 



,368 



0,000 



,96a 



,975 



.680 



Cataoory Marginal AgraamanM 
Cohan Kappa I 



er|c 



Gil 



612 



' Table 326 

Phase IV «• IGE Evaluation Project 
Descriptive study - spring 1978 

Reliability on intarectlons 
Observer UQ Expert 80 
School BOX Grade 5 
Week 9 Day * 
OMP 



Detel 06-25-79 
•program! REi.P'S 
Data Fliei PEID1 



Category chosen by Observer 10 



Category choten 
by cxpart 80 



Cat, 

ft 



10 il 











Teroet to Teacher 


V 


♦ 
♦ 


0 


Target to Other Adult 


2 


♦ 
♦ 


0 


Target to Student 


3 


♦ 
♦ 


0 


Target to Tchr i Grp 


<* 


* 


0 


Teacher to Target 


5 


♦ 

♦ 


0 


Other Adult to Target 


6 


♦ 


0 


Student to Target 


T 


♦ 

' ♦ 


0 


Teacher to Group 


e 


♦ 


l 


j Other Adul t to Group 


9 


♦ 
♦ 


0 


(student, to Group 


10 




0 


N9*1 o# the above w 


n 


♦ 
♦ 


0 


Marginal Totals 


♦ 


l 











i V 



0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 



0 

0 

2 

0 

0 

0 

0 

0 

0 

0 

2 



0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 



0 
0 

o * 

0 
0 
0 
0 
0 
0 
0 

I 



0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 



0 
0 
0 
0 
0 
0 
2 

1 

0 
0 
0 



0 
0 
0 
0 
2 
0 
0 
8 



16 



0 
0 
0 
0 
0 




0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 



0 
0 
0 
0 

1" 

0 
0 
3 
0 

61 

68 



Marginal 
Totals 



++++++++ 

0 

0 

2 

0 

3 

0 

2 
13 

0 

0 
TO 

♦♦♦♦♦♦♦♦ 
90 



uight«t 
Extension 

Agreement Agreement o' Keppe 



Nominal Marginal 



ltOOO 

0.000 

1.000 
*618 

,871 



0.000 
.800 

• 333 

• 667 
.613 

• 929 



I. 000 
• •Oil 

1.000 

• 532 

• 837 



613 



614 



Coh«n»i KMMt t**0 



Table 327 

Phase IV - IGE Evaluation Projtet 
Detcrlptlve Study • Spring 1978 

Reliability on interactions 
Obierver 10 Export 90 
School 801 Gradft 2 
Week 9 Day 4 

MORSO 



Data! 06-25-79 
Programi RELP9S 
Data f I I a I REID1 



Category choien by obierver 10 



Category choien Cat* 
by OPart 90 * 



Marginal Toteji 



1 29 



10 11 



Target to Teachar 


1 ♦ 


0 


0 


0 


0 


0 


0 


0 


1 


0 


0 


0 


Target to Ot*er Adult 


2 ♦ 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 


Target to Student 


3 ♦ 


0 


0 


1 


0 


0 


0 


1 


0 


0 


0 


0 


Targec to Tchr i Orp 


4 ♦ 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 


1 


Teacher to Target 


5 ♦ 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 


Other Adult to Target 


6 ♦ 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 


u 


Student to Target 


T ♦ 


0 


0 


0 


0 


0 


0 


0 


1 


0 


0 


0 


Teacher to oroop 


8 ♦ 


2 


0 




0 


0 


0 


0 


23 


0 


0 


3 


Other Adult to Group 


9 ♦ 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 


Student to Group 


10 ♦ 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 


1 


hone o# the above 


11 ♦ 


i 


0 


0 


0 


0 


0 


0 


4 


0 


3 


24 



29 



Marginal 
Total • 

♦♦♦♦♦♦♦♦ 

1 

0 

2 

1 
0 
0 
1 

28 

0 
1 

32 

♦♦♦♦♦♦♦♦ 
♦♦♦♦♦♦♦♦ 



LlghtM 

Nominal Marginal Extem|on 
Agreement AOreawent ©# Kappa 



0.000 

• 500 
0.000 



OtOOO 

• 821 

0t000 

• rto 



,333 



,500 
0,000 



ItOOO 

• 96* 

• 333 
,904 



• ,048 

,492 
0,000 



«,0iB 
.681 

• •048 
,884 



9 

ERIC 



category Marginal Agree*antl 
Cohen*! wappai ,844 



0 ] (j 



Table 328 

Phase IV - IOE Evaluation Project 
Descriptive Study - Spring 197B 

Reliability on Interactions 
Obierver 20 Expert 90 
, School 801 erode 2 
Week 9 Day * 
H0R5D 



Oatel 06-25*79 
Proorami relpos 
Date PI lei RELDi 



Category choten Cat. 
by Expert 90 # 



Category choten by Obierver 20 



G 



Herglnel ToteH 



O 20 



t; ] v 



io n 



Target to Teacher 


1 ♦ 


0 


0 


0 


0 


0 


0 


0 


0 


0 


1 


0 


Target to Other Adul t 


2 ♦ 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 


Targat to Student 


3 ♦ 


0 


0 


2 


0 


0 


0 


0 


0 


0 


0 


0 


Target to Tchr L Orp 


4- ♦ 


0 


0 


0 


0 


0 


0 


ft 


1 


0 


0 


0 


Teacher to Target 


5 ♦ 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 


Other Adult to Target 


6 ♦ 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 


Student to Target 


7 ♦ 


0 


0 


0 


0 


0 


0 


0 


1 


0 


0 


0 


Teacher to orogp 


8 ♦ 


1 


0 


1 


0 


0 


0 


0 


15 


0 


0 


11 


Other Adult to orouP 


9 * 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 


Student to Group 


10 ♦ 


0 


0 


0 


0 


0 


0 


0 


1 


0 


0 


0 


Njne ol the ebove 


11 ♦ 


0 


0 


2 


0 


0 


0 


0 


2 


0 


3 


25 



38 



category Marginal Agreement! *46§ 
Cohen Keppei #397 



Merglne I 
Toteit 

♦♦♦♦♦♦♦♦ 

1 

0 

2 

1 
0 
0 

1 

28 

0 
1 

32 

♦♦♦♦♦♦♦♦ 
86 

♦♦♦♦♦♦♦♦ 



Nominal 
Agreement 



0*000 



Lights 
Marginal Extension 
Agreement of Kappa 



l •ooo 

0,000 



0*000 
• 536 

0*000 
.Til 



1*000 

• 400 

0,000 



0*000 

• 860 
,M0 



,018 



l»ooo 

0,000 



0*000 
,334 

• ,048 
.•19 



9 

ERIC 



/ 



Table 329 

Phaia IV - IOE Evaluation Projact 
Daicrlptiva Study • spring 1976 

RaUablllty on interaction! 
Obtarvar 40 Expart ©0 
School 801 Orada 2 
Waak 9 Day 4 
WDRSD 



Datal 06-25*79 
Program) REIP9S 
Data f| 1*1 RELD1 



Catagory c*otan 
by Export BO 



Cat, 

a 



Catagory choaan by obtarver 40 
i 2 3 4 5 6 7 B < 



Marginal Total! 



10 



Targat to talschar 


♦ 

1 ♦ 
♦ 


3 


0 


0 


0 


0 


0 


0 


1 


0 


0 


Targat to Othar Adul t 


8 ♦ 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 


Targat to Studant 


♦ 

3 ♦ 
♦ 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 


Targat to ToHr % Orp 


4 ♦ 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 


Taachar to Targat 


♦ 

S ♦ 


0 


0 


0 


0 


2 


0 


0 


1 


0 


0 


Othar Adult to Targat 


♦ 

* 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 


Studant to Targat 


T ♦ 


0 


0 


1 


0 


0 


0 


0 


0 


0 


0 


TaacKar to oroup 


e ♦ 


0 


0 


0 


0 


0 


0 


0 


T 


0 


0 


Othar Adul t to oroUP 


9 ♦ 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 


Studant to oroup 


10 ♦ 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 


Nona o# tha abova 


11 ♦ 


0 


0 


0 


0 


0 


0 


3 


5 


0 


G 



1* 



11 

0 
0 

1 

0 
0 
0 
0 

1 

0 
0 
39 

3T 



Marginal 
Totals 

♦♦♦♦♦♦♦♦ 

4 

0 

I 

0 
3 
0 
1 
8 
0 
0 
43 

40 

♦♦♦♦♦♦♦♦ 



Nominal Marginal 
Agraemant Agraemant 



,750 



0*000 

,667 

0,000 
,676 

#•14 



,760 

1,000 

,667 

,333 
,671 

,860 



ug*t«i 

Extant |on 
of .Kappa 



i737 



• •017 
,665 

• ,053 
,637 

tfil5 



ERLC 



(11!) 



Catagory Marginal Agraamanti f 6?7 
CoHan»t Kappat t»64 



6 2 0 



Table 330 

Phase IV - 10E Evaluation ^ojtct 
DaicrlPtlve Studv * SPMnO 1978 

Reliability on interaction* 

Observer f<J Expert 90 

School 801 Grade 2 

Week 9 Day * 



oc 



Datel 06-25*79 
Program! HELP9S 
Data M It I ^CtOl 



Category choten 
by E*part 90 



Target to Teacher 



Cat • 



♦ 

1 ♦ 



target to Other Adult 2 ♦ 
Target to Student 



Target to TcHr l orp 
Teacher to Target 



3 ♦ 

4 ♦ 
♦ 

5 ♦ 

♦ 

Other Adult to Target 6 ♦ 
student to Target T ♦ 

Twecher to Group * ♦ 

♦ 

Uthar Adult to oroup 9 ♦ 
Student to Group 10 ♦ 

hone of the ebove 11 • 



0 
0 
0 
0 
1 



0 

0 

0 

0 

0 

0 

0 

0 



Marginal Total* ♦ 1 



1>2I 



Category choaen by ObMrvtr 
3 h 8 6 7 0 



wergl nal 
Toteia 



0 
o 
1 
0 
0 
0 
0 



0 

0 

0 

0 

0 

0 

0 



0 
0 
0 
0 

0 
0 
0 



0 

0 

0 

0 
0 
0 
0 
0 



9 10 11 

▲ 

ft 0 



0 

1 

«f 

0 

1 



0 
0 
0 
0 
0 

ft 

9. 
0 



0 


0 


* 


♦ 


0 








♦ 




0 


0 


0 
















0 


0 


3 




a 








♦ 




0 




0 


♦ 


0 








* 




0 


0 


0 




i 








♦ 






0 


0 




0 












0 


0 


3 




J 












0 


0 


0 


♦ 


8 








♦ 






0 


0 


♦ 


0 












0 


0 


0 




0 








♦ 




0 


0 


70 


♦ 


73 


















♦ 




0 


0 


7S 




©4 



Catooory Marolnul Agr«a»«ntl **9* 
Oohen'f Kappal 



Nominal Marolnal 
Agreewant AOrtament 



UQhttt 
Extaniion 
of Kappa 



,300 



0,000 



0*000 



.too 



OtOOO 



,333 1*0*0 
1,000 uooo 



,973 



,1T0 

0.000 

,309 
1*000 



Table 331 

Phase IV - !0E Evaluation project 
DetcrlPtlve Study - spring 1978 

pai lability on interaction* 
Observer 20 Expert 90 
School SOI Grade 2 
week 9 Day * 
OMP 



Datal 06-25-79 
Programi REI.P9S 
Data ri lat ,REL01 



Category choien 
by expert 90 



Cat, 
n 



Marginal Totan 



Category ehoeen by Obaerver SO 

3 * 5 6 T 8 9 10 U 



Marginal 
Totals 

♦♦♦♦♦♦♦♦ 



t ightto 

Nominal Marginal Extension 
Agreament Agreement of Kappa 



Taraat to Taachar 


1 ♦ 


0 


0 


0 


0 


0 


0 


0 


Target to Other Adult 


2 ♦ 


0 


0 


0 


0 


0 


0 


0 


Targat to Studant 


3 ♦ 


0 


0 


* 


0 


0 


0 


0 


Targat to Tchr I orp 


4 ♦ 


0 


0 


0 


0 


0 


0 


0 


Taachar to Target 


5 ♦ 


1 


0 


0 


0 


0 


0 


0 


Othar Adult to Target 


6 ♦ 


0 


0 


0 


0 


0 


0 


0 


Studant to Targat 


T ♦ 


0 


0 


0 


0 


0 


0 


0 


Taachar to oroup 


fi ♦ 


0 


0 


0 


0 


0 


0 


0 


Othar Adult to oroup 


9 ♦ 


0 


0 


0 


0 


0 


0 


0 


Studant to oroup 


10 ♦ 


0 


0 


0 


0 


0 


0 


0 


Nona of tha above 


11 ♦ 


0 


0 


2 


0 


0 


0 


0 



0 
0 
0 
0 
0 
0 
0 

1 

0 
0 
0 



0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 



0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 



♦♦♦♦♦♦♦♦ 



0 


♦ 0 ♦ 




0i000 




0 


♦ 0 ♦ 








1 


♦ 6 ♦ 


,800 


,833 


,785 


0 


♦ 0 ♦ 








0 


♦ 1 ♦ 


0i000 


0,000 


0,000 


0 


♦ 0 ♦ 








3 


♦ 3 ♦ 


0,000 


0,000 


0,000 


1 


♦ 2 ♦ 


♦ 600 


,500 




0 


♦ 0 ♦ 








0 


♦ 0 ♦ 








71 


♦ 73 ♦ 


,973 


,961 


•m 












7* 


6 8% ♦ 









9 

ERIC 



623 



Category Marginal Agreement! ,3St 
Cohen*a Keppel ,S*S 



624 



00 



Table 332 

0 

Phata IV • I Ot Evaluation Projact 
DascrlPtlva Study ■ SPMno 1978 

Rai lability on Intaractlont 
Obtarvar tyQ Expart 80 
School 801 Grada t 
Waak 9 Day * 
OMP 



4-- 



Datal 06-25*79 
programl PELP96 
Oata F RClDl 



Catagory choian 
by Export *0 



Catagory chotan by obsa?*var 10 
3 4 5 6 7 8 




Taroat to Taacher 


1 < 


► 1 


0 


0 


Taroat to Othar Adult 


2 < 


► 0 


0 


0 


Taroat to Studant 


3 i 


► 0 


0 


2 


Taroat to Tchr I orp 


4 t 


> 0 


0 


0 


Taochar to Targat 


5 


» 0 


0 


0 


Othar Adult to Taroat 


6 


♦ 0 


0 


0 


Studant to Taroat 


7 


♦ 0 


0 


1 


Taacnar to oroup 


8 


♦ 3 


0 


0 


Othar Adul t to oroup 


9 


♦ 0 


0 


0 


Studant to Oroup 


10 


♦ 0 


0 


0 


Nona ol tha abova 


11 


♦ 0 


0 


8 












Marginal Totals 


♦ 3 


0 


11 



Marginal 
Totals 



025 



0 
0 
0 
0 
0 
0 
0 
0 
0 
0 
0 



0 


0 


0 


1 


0 


0 


0 ♦ 


a 


♦ ,500 


0 


0 


0 


0 


0 


0 


0 ♦ 


0 




0 


0 


0 


0 


0 


0 


0 ♦ 


8 


♦ 1,000 


0 


0 


0 


0 


0 


0 


0 ♦ 


0 




1 


0 


0 


1 


0 


0 


0 ♦ 


a 


♦ ,500 


0 


0 


0 


0 


0 


0 


0 4 


0 




0 


0 


0 


0 


0 


0 


1 4 


2 


♦ 0,000 


0 


0 


0 


3 


0 


0 


* 4 


9 


♦ ,333 


0 


0 


0 


0 


0 


0 


0 i 


► 0 




0 


0 


0 


0 


0 


0 


0 < 


► 0 




0 


0 


1 


7 


0 


0 


<*5 4 


► 61 





12 



50 



LlflHtta 

Now I na I Maro I na | Extant | on 
Aoraamant Aoraamant Kappa 



78 

♦♦♦♦♦♦♦> 



Cataoory Marginal Agraamantl #570 
Cohan* I M8PP8I #899 



• 667 

• 162 

• 500 

• 500 

• 750 

• 820 



• 480 
1,000 

• ^94 

• •013 

• 212 

• 869 



626 



Category choien 
by Expert 90 



Target to Teacher 

Target to Other Adult 2 

Target to Studant 3 

Target to Tchr I Orp * 

Teecher to Target * 

Other Adult to Target 6 

Itudent to Target ▼ 

Teacher to oroup 8 

Other Adult to oroup 9 

Student to Group 10 

hone o# the above U 



♦ 
♦ 
♦ 
♦ 
♦ 
♦ 
♦ 
♦ 

♦ 
♦ 
♦ 
♦ 
♦ 

♦ 
♦ 
♦ 
♦ 



Marginal Total* 



Table 333 

Pheu> IV - I0E Evaluation Project 
Descriptive Study ^spring 1976 

Reliability on intfcractlont 
Observer 10 Expert 90 
School BOl Orade 5 
Week 9 Oay 5 
DMP 



Category choaen bv Obiarver 10 





Datal 06-25-79 
Program! RELP9S 
Oeta fl lei RElDl 



Marginal 
Totaia 

♦♦♦♦♦♦♦♦ 



o 
l 

0 
0 
0 

V 

2 

0 

0 

1 

66 
90 

♦♦♦♦♦♦♦♦ 



Nominal 
Agreement 



Ltght it 
Marginal EKtantion 
Agreament c# Kappe 



0,000 



0, 000 



Of 000 
,96* 



0t 000 



0,000 



0,000 



0,000 
,966 



0,000 



0,000 



0,000 
• ,046 



627 



Category Marginal Agreementi c 193 
Cehen*a Kappal • •009 



628 



TaMe 134 



Phait ZV - IOK Evaluation Projact 
OticriPtlvo Study * SPnng 197i 

Reliability on Interactions 
Obtarvar ?0 Ewptrt 90 
School 801 Oredt 5 
weak 9 Day 6 

DMP 



Oatal 06-25*79 
Program! Rtl P9S 
Data Fllll 



Catagory cKotan Cut, 
by ExP«rt 90 " 



Category chooan by Observer 20 
1 2 3 * 5 A 7 8 1 



10 



1 1 



Mar g I na I 



Nominal 



Marginal 



Light*! 
Cxt«"t »on 



Total • Agreement Agreement o# Kappa 



Tergtt *c Teacher 


i * 


► 0 


0 


0 


o 


o 


o 


0 


0 


0 


0 


0 


4 0 


4 

4 


Target to Other Aciu 1 1 


? 


* 0 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 


4 0 


4 


Targat to Student 


3 


► 0 


0 


0 


0 


o 


0 


0 


0 


0 


0 


1 


4 1 


4 

4 


Turgat to Tchr orp 




♦ 0 


0 


0 


0 


0 


r 


0 


0 


0 


0 


0 


4 0 


4 

* 


Teacher to Targat 


5 


♦ 0 


0 


0 


0 


0 


0 


0 


0 


c 




0 


4 0 


4 
4 


Other Adult to targat 


6 


♦ 0 


0 


0 


c 


G 


0 


0 


0 


0 


0 


0 


4 C 


4 
4 


Studant to Targat 


7 


♦ 0 


0 


0 


0 


0 


0 


i 


0 


0 


0 




4 2 


4 
4 


Taachar to oroup 


0 


♦ 0 


0 




0 


0 


0 


c 


0 


0 


0 


0 


♦ 0 


4 
4 


Othar Adult to oroup 




* J 


0 


0 


«j 


n 


0 


0 


(/ 


0 


0 


0 


* 0 


4 
4 


Student to oroup 


10 


♦ 0 


0 


0 


0 


0 


0 


0 


0 


0 


0 


1 


♦ » 


4 


Nona c<> th© abovo 


n 


4 \ 


0 


0 


0 


0 


o 


0 


0 


fl 


0 




4 *6 


4 
4 








4 4*4 + 


+ 444 4' 


4 4 4 4 « 


► ♦ * 4 < 


* * * * 4 * * 


* * 4 


I k 4 4 4> <i 4 4 


> * * 4 ♦ 


' 4 


v 4 4 4 4 4 4 4 


♦ 444 


Marojnis! Tota 


i • 


* 


0 


0 


0 


0 






n 


0 


o 


*s 


* «s 


4 

# 








****** 


4 4 4 :< ♦ 


* 4 ♦ *? t- 


4 4 4 4* 


1 4 4 4 i 




4 * * 


4**4*4*4 


4*4 * 


#*444*4*444 


444 * 



4 >444444* 
4 



0.000 



>S00 



0,000 
• 9M 



0i000 



0,000 



• 500 



0.000 
• •77 



0.000 



>*94 



0.000 
.♦77 



category Marginal Agraamant i * 



ERIC 



t 



Table. 33^ 

Phase IV - I0£ Evaluation Project 
Descriptive Study « SPMng 1978 

Reliability on interaction! 
Observer 30 Expert 80 
School 601 Grade 5 
week 9 Day 5 

DM? 



Category chosen Cet» 
by ExPa^t 80 * 



Category chosen by Observer 30 



10 



11 



Target to Teacher 


1 




0 


0 


0 


0 


1 


0 


0 


1 


0 


0 


1 


Target to Other Adult 


2 


♦ 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 


Terget to Student 


3 


♦ 


0 


0 


1 


0 


0 


e 


0 


0 


0 


0 


0 


Target to Teh* i orp 


* 


♦ 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 


Teacher to Target 


5 


* 


1 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 


Other Adult to Target 


6 




0 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 


Student to Target 


7 


♦ 
♦ 


0 


0 


0 


1 


0 


0 


0 


0 


0 


0 


0 


Teacher to oroup 


8 


♦ 


\J 


0 


0 


1 


0 


0 


0 


6 


0 


0 


5 


Other A \ilt to Oroup 


9 


♦ 


0 


0* 


0 


0 


0 


0 


0 


0 


0 


0 


0 


Student to orogp 


10 


♦ 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 


hone o# the above 


11 


♦ 

4 


0 


0 


T 


s 


1 


0 


0 


2 


0 


0 


60 



4 
♦ 

Merginal Totelt ♦10942009 
♦ 



66 



category Marginal Agreewenn ,371 
Cohen's Keppei #389 



i;;n 



9 

ERIC 



Datel 06*25-79 
Program R£iP9S 
Data f | lei REtDl 



Mergl nal 
Total! 

3 

0 
1 

0 

1 

0 

) 

1* 

0 
0 
72 

90 



Now I n* | 



Marginal 



Light U 
Extend | on 



Agreement Agreement of Kappa 



0,000 

1,000 

0,000 

OiOOO 
,500 



,833 



• 333 

,125 
0,000 
,500 

0,000 
,75C 



,917 



• .on 
i,o<to 

• ,023 
0,000 

***** 



,375 



(X 



U> 

CO 



Table 3Jft 

Phut* IV - IOE Evaluation Project 
Deicrlptlva Study • Spring 1978 



Ril (abl I Itv on interaction* 
Observer AO Expert 80 
School 6 01 Grade 5 
Weak 9 Day 5 
DMP 



Date! 06»25»T9 
Programl REIP9S 
Date F||e| REtDl 



Category eho*en by Obierver AO 



Catagory chosen Cat« 
by Expert 80 * 



10 



Marginal Total* 



Marg I na I 
Total s 

♦♦♦♦♦♦♦♦ 



Targat to Taachar 


1 ♦ 


1 


0 


0 


0 


0 


0 


0 


2 


0 


0 


0 i 


> 3 


Targtt to Othar Adult 


2 ♦ 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 4 


> 0 


Targat to Student 


3 ♦ 


0 


0 


1 


0 


0 


0 


0 


0 


0 


0 


0 < 


> 1 


Targat to Tehr 1 Orp 


4 ♦ 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 4 


► 0 


Teacher to Tergat 


5 ♦ 


0 


0 


0 


0 


1 


0 


Q 


0 


0 


0 


0 4 


► 1 


Othar Adult to Target 


6 ♦ 


0 * 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 i 


► 0 


Student to Target 


7 ♦ 


0 


0 


0 


0 


0 


0 


1 


0 


0 


0 


0 < 


► 1 


Teecher to orogp 


0 ♦ 


0 


0 


0 


0 


0 


0 


1 


T 


0 


0 


4 4 


> 12 


Other Adult to orouP 


9 ♦ 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 4 


► 0 


Student to Group 


10 ♦ 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 4 


> 0 


Nona c# the ebove 


u ♦ 


i 


0 


3 


0 


1 


0 


2 


13 


0 


0 


*» 4 


► 72 



2040*0* 22 00 5* 



♦♦♦♦♦♦♦♦ 
90 

♦♦♦♦♦♦♦♦ 



Catagory Marginal Agraamantl 
Cohan'* xeppai ,J89 



,498 



9 

ERLC 



Llflht 

Nominal Marginal Extension 
AOraement Aflreemant o# Kappa 



333 



,667 



,316 



1,000 

1 #000 

1,000 
,583 

„T22 



,250 

,500 

,260 
,545 

• T78 



1,000 
1*000 

1,000 
,449 

,285 



fi.'M 



Table. 337 

Pheae IV - IOE Evaluation project 
Deoerlptlve Study - spring 19T8 

Pal labl I I tv on interactions 
C-baervar 10 Expert 90 
School 801 Grade 2 
Hack 9 Day 5 
MORSO 



Oatel 06-25-T9 
Program! REL.P9S 
Data Fl l«l REtO\ 



Category cho«an 
bv expert 90 



Category cho»en by Obiervar 10 
6 7 8 9 




Marginal 
Total* 



UlOht'l 

Nominal Marginal Extent ion 
agraament agreement o* Kappa 





4 










7aroat to TaacKar 


1 


♦ 


0 


0 


0 


Target to Othor Adult 




♦ 

♦ 


0 


0 


0 


Taroat to student 


3 


♦ 


0 


0 


I 


Taroat to Tchr I <lrp 

r 


4 


♦ 


0 


0 


0 


Taachar to Taroat 


5 


♦ 
♦ 


0 


0 


0 


Ct*ar Adult to Taroat 


6 


4 
♦ 


0 


0 


0 


3tuda*t to Taroat 


7 


♦ 
♦ 


0 


0 


1 


Taachar to orouo 


8 


4 
♦ 


0 


0 


0 


Cthar (iu 1 1 to Group 


9 


♦ 
♦ 


0 


0 


0 


Studartt to oroup 


10 


♦ 
♦ 


0 


0 


0 


Nona o# tha abova 


11 


♦ 


1 


0 


5 


Haro^nal Totalt 


♦ 

4 


I 


0 


T 



0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

1 



0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

z 



0 

0 

0 

0 
0 
0 
0 
0 

0 
0 



0 
0 
0 
0 

0 
0 
0 
0 
0 
0 



0 
0 
0 
0 
0 
0 
0 
XI 
0 
0 
4 

is 



0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 



0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

1 













0 


♦ 0 ♦ 




0,000 




0 


♦ 0 • 








a 


♦ 3 ♦ 


• 333 


,429 


.254 


X 


♦ I ♦ 


0,000 


1 ,000 


-.01B 


0 


♦ 0 ♦ 




0*000 




0 


♦ 0 ♦ 








1 


♦ 8 ♦ 


0,000 


0,000 


0*000 


1 


♦ \2 ♦ 




t soo 


i69i 


0 


a 0 ♦ 








0 


♦ 0 ♦ 




o«ooo 




34 


♦ *a ♦ 


iTOi 


#•13 


,187 












39 


a *ft ♦ 



















(>.{.) 



catagory Marginal agraementl .3*0 
Ct h*n*a Keppai «*2t 



l!3K 



(.>- 



EMC 



Table 338 

Phase IV - I GE Evaluation Projtct 
Descriptive Study - spring 1978 

Reliability on Interactions 
Observer 20 Expert ^0 
School 601 Grade 2 
Week 9 Day 5 
WDRSO 



C 



Datel 06*25*79 
Progre*l RELP9S 
Deta f | lei REUOl 



Category chosen by observer 30 



Category chosen Cet, 
by Expert 90 * 



Naroinel Totals 



0 0 



1* 



10 11 



Target to Teacher 


\ ♦ 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 


Target to Other Adul t 


2 ♦ 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 


Target to Student 


3 ♦ 


0 


0 


2 


0 


0 


0 


0 


0 


0 


0 


1 


Target to Tchr I Orp 


A ♦ 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 


1 


Teacher to Target 


5 ♦ 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 


Other Adult to Terget 


6 ♦ 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 


Student to Target 


7 ♦ 


0 


0 


1 


0 


0 


0 


0 


0 


0 


0 


i 


Teacher to Group 


fi ♦ 


0 


0 


0 


0 


0 


0 


0 


10 


0 


0 


a 


Other Adult to oroup 


9 ♦ 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 


Student to Group 


10 ♦ 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 


None of the above 


11 ♦ 


0 


0 


3 


0 


0 


0 


0 


4 


0 


1 


40 



AS 



Marginal 
Totals 

♦*♦<►♦♦♦♦ 

0 

0 

3 

1 

0 

0 

a 

12 

0 

0 
AS 

6ft 



Light's 

Nominal Marginal Extension 
Agreement Agreement o* Keppe 



,667 
0,000 



0c000 
• 833 



} S33 



,soo 

0,000 



0,000 
,SB7 

0*000 
,938 



,633 
0,000 



0,000 
,7ftS 



,#76 



Category Marginal Agreement! ,3a* 
Cohen's xappei ,t*0 



Table 339 

Phot* IV «• IOC Evaluation Project 
DatcrlPtlva Study - spring 1978 

Pallablllty on intaractlont 
Obtarvar 30 Expart 80 
School 801 Orada 2 
Waek 9 Day 9 

WORS0 



Data I 06-25-79 
Programl REL**** 
Data r\ lai RELDl 



Cataoory eho«en Cat f 
by ExPart 80 * 



Cataoory choean by Obtarvar 30 



6 



10 U 



Targat to Taachar 


1 « 


► 0 


0 


0 


0 


0 


0 


0 


0 


0 


0 


1 


Targat to Othar Adult 


2 


> 0 


0 


0 


0 


0 


c 


0 


0 


0 


0 


0 


Taroat to studant 


3 


* 0 


0 


2 


0 


0 


0 


0 


0 


0 


0 


0 


Targat to Tchr i Orp 


4 


♦ 0 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 


Taachar to Targat 


S 


* 0 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 


Othar Adul t to T«rgat 




♦ 0 


0 


0 


0 


0 


0 


0 


0 


0 


0 


n 


Studant Targat 


7 


♦ 0 


0 


1 


0 


0 


0 


1 


0 


0 


0 


0 


Taachar to group 


6 


♦ 0 


0 


0 


0 


0 


0 


0 


3 


0 


0 


7 


Othar Adult to oroup 


9 


♦ 0 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 


Studant to Group 


10 


♦ 0 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 


hona of tha abova 


11 


♦ 1 


0 


1 


0 


0 


0 


1 


0 


0 


0 


42 






♦♦♦♦♦♦♦♦ 












♦ ♦♦♦♦ 










Marginal Totals 


♦ 1 


0 


4 


0 


0 


11 


2 


3 


0 


0 


so 



light* • 

Marginal Nominal Marginal Ext«*t|on 
Totals Agraamant Aarftarnant of Kappa 



1 
0 
2 

0 
0 
0 
2 
10 

0 
AS 

80 

♦♦♦♦♦♦♦♦ 



0i000 



1*000 



1 ,000 



9 S00 



• 500 
#300 



1 o000 
,300 



,933 



,900 



,017 



1.000 



#483 
• 263 



**Q0 



CUD 



catagory Marginal Agraawantl .740 
Cohanifl xappai •*** 



wo 
H 



Table 340 

Phase XV • IGE Evaluation Project 
Descriptive study - spring 1976 

Reliability on Interactions 
Observer 40 Expert 00 
School 801 Grade 2 
week 9 Day 6 

WDRSD 



Oatel 06-25^79 
Prooraml RELP9S 
Date Mlei REL01 



Category c osen by Observer *0 



Category chosen Cat# 
by Expert 80 * 



10 11 



Marginal Totals 



hi 



Category Marginal Agreementi #6*6 
CoHents Kappai • IS* 



Marginal 
Totals 

♦♦♦♦♦♦♦♦ 



Targat to Teacher 


1 ♦ 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 


1 4 


> . 1 


Target to Other Adult 


2 ♦ 


0 


0 


0 


0 


0' 


0 


0 


0 


0< 


0 


0 < 


> 0 


Targat to Student 


3 ♦ 


0 


0 


0 


0 


0 


0 


1 


0 


0 


0 


1 4 


> 2 


Targat to Tchr t orp 


* ♦ 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 4 


> 0 


Teachar to Target 


5 ♦ 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 4 


> 0 


Cther Adult to Targat 


6 ♦ 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 4 


► 0 


Student to Targat 


T ♦ 


0 


0 


1 


0 


0 


0 


1 


0 


0 


0 


0 4 


► 2 


Teacher to Oroup 


8 ♦ 


0 


0 


0 


0 


0 


0 


0 


T 


0 


0 


3 4 


► 10 


Other Adult to oroup 


9 ♦ 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 4 


► 0 


Student to Oroup 


10 ♦ 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 < 


► 0 


Hone ot the above 


11 ♦ 


0 


0 


1 


0 


0 


0 


0 


2 


0 


1 


M 4 





♦♦♦♦♦♦♦♦ 
60 



tlghtte 

Norn | na I Mergl nai Ex tens ion 
Agreement Agreement of Kappa 



0,000 



0«0D0 



• 50 0 
9 TO0 



0,000 



loOOO 



ltOOO 
|90C 

o#ooo 

• ITS 



612 



0,000 



.034 



«463 



.619 



Table 341 

Phasa Iv - IOC evaluation Project 
Deacrlptiva study * spring 1976 

Reliability on Interaction* 

Observer 10 Expert 90 

School 801 Orada 3 

week 9 Day 5 

OMP 



Oatal 06-25*79 
program! RELP9S 
Oat* Fl Id RELD1 



Category choien 
by Expert 90 



Cat, 
a 



Category choten by obttrvtr 10 
3 4 5 6 7 8 



9 10 U 
,♦♦♦♦♦♦< ♦♦♦♦♦♦ 



Hargl na | 
Total t 

♦♦♦♦♦♦♦♦ 



Target to Taacher 


1 < 


► 1 


0 


0 


0 


0 


0 


0 


0 


0 


0 


1 + 


I « 


Terggt to Other Adult 


2 < 


► 0 


0 


0 


0 


0 




0 


0 


0 


0 


0 ♦ 


0 < 


Target to Student 


3 < 


> 0 


0 


0 


0 


0 


0 


0 


0 


0 


0 


c ♦ 


0 < 


Targat to TcHr i Orp 


4 « 


► 0 


0 


0 


0 


0 


0 


0 


0 


0 


0 


i ♦ 


1 < 


Teacher to 7argat 


5 


» 0 


0 


0 


c 


0 


0 


0 


0 


0 


0 


0 * 


0 < 


Other Adult to targat 


ft 


♦ 0 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 ♦ 


0 * 


Studaftt to Target 


7 


♦ 0 


0 


0 


0 


0 


0 


\ 


0 


c 


0 


1 « 


1 < 


Teacher to oroup 


8 


♦ 0 


0 


0 


0 


0 


0 




3 


0 


0 


I « 




Other Adult to oroup 


9 


* 0 


0 


0 


0 


0 


(5 




0 


0 


0 


0 4 


■ 0 


Student to oroup 


10 


• 0 


0 


0 


0 


0 


n 


O 


0 


0 


0 


0 < 


► 0 


None o# the ebove 


U 


♦ 1 




i 


0 


0 


0 


O 


? 


0 


1 


70 < 


> 75 












Marginal Totaia 




0 


1 


0 


0 


0 


i 




0 


t 


7* 4 


► 6* 






♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 


4 ^ 





Nominal Marginal 



Lights 

Extent I on 



Agreement Agreement of Koppe 



• 500 



0,000 



,S0Q 
,7*0 



,933 



1,000 

0,000 
0,000 



,500 
,800 

0,000 
,987 



,488 



0,000 



,494 
.734 



1*40 



category M*-ginai Agreement! ,4y3 
Cohen* * Keepai ,**9 



I 0 



er|c 



Table 342 

Phatt IV ~ IDE Evaluation Projact 
OatcrlPtlva Study * spring 1978 

Reliability on interactions 
Observe** 20 Expert 90 
School 801 orada 2 
Weak 9 Day 8 

OMP 



Datai 06-25-79 
Program REIP9S 
Data F I I • I REL01 



Category chotan by Obsarver 20 



Cetagorv chonen Cat# 
by EMPar* 90 * 



tarwat to Taachar 1 

Targat to ot^ar Adul t 2 

Tercet to Student 3 

Targat to Tehr i orp * 

Taachar to Targat 8 

Othar Adult to Targat 6 

Studant to Targat T 

Taachar to Group 6 

Othar Adult to oroup 9 

Studant to Group 10 

Mone of tha abova 11 

Marginal Totals 
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2 


3 
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5 


6 


T 
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9 


10 
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♦ 0 ♦ 
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0 


0 
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0 




♦ 0 ♦ 


♦ 0 


0 


0 


0 


0 


0 
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0 


0 
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0 
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0 


0 
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0 
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0 


0 


0 
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♦ 0 


0 


0 


0 


0 


0 


0 


0 


0 


0 




♦ 0 ♦ 


♦ 0 


0 


0 


0 


0 


0 


0 


0 


0 


0 




♦ 0 ♦ 


♦ 2 


c 


1 


0 


1 


0 


1 


0 


0 


0 


TO 


♦ T8 ♦ 


♦ 3 


0 


2 


0 


1 


0 


1 


3 


0 


0 
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Table 343 

Phe.. IV - !0E evaluation "oj.ct 
Oaiclptlva Study - spring 1978 

Hal tab) I 1 1 v on interaction. 
Ob«orvar 30 Fxpert 80 
School B01 Orada 2 
weak 9 Day S 
DMP 



Oatal 06-26-79 
proorami HELP** 
Oata F| lal RKUD1 



Catagory choian 
by EKP«<*t 80 



Cat* 



Targat to Taaehar 
Targat to Othar Adult 
Targat to Studant 
Taroat to Tchr i orp 
Taaehar to Targat 
Othar Adult to Targat 
Studant to Targat 
Taachar to oroup 
Othar Adult to oroup 
Studant to ©roup 
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Category chotan by obsarvar 30 
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0 
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0 
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0 


0 


0 
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0 
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0 
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0 
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0 


0 
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0 
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0 
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0 
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0 
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0 
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♦ 










0 


0 


2 
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0 
0 
0 
0 
0 
0 

1 
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0 

0 

0 
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0 

0 

0 

0 

0 

0 
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0 


0 
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0 


0 
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0 


0 
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0 


c 


0 


* 
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♦ 




0 


0 


0 


0 
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♦ 




0 


0 


0 


1 


♦ 
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5 


0 
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0 
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0 
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0 


0 


0 


0 
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0 
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♦ 
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Tab If? 344 

Pho»<, :\ ■ Trr rtion Pmj^rt 

oaecrlptiva Study - spring 19 fa 

^•liability on interactions 

Obaarvar 40 Export 80 

School 601 Grada 2 

Weak 9 Day 5 

PMP 



Data* 06*26-T9 
program &EIP9B 
Data fi ia| RELD1 



Catagory cKofan by Obsarvar 40 



Catagory chotsn 
by CKPart 80 



Cat o 
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Targat to Taaehgr 
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♦ 
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♦ 

♦ 
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0 
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0 3 9 0 % 

Catagory Marginal Agrt*a**nti f M* 
Cohan* 8 Kappal 
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APPENDIX C 



Contingency Tables for each Set of Observation Categories 
for Pairs of Observers for the On-site visits 
to the WDRSD and DMP Schools 
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Appendix C 

Included in this appendix are: 

Tables 345 - 350 Observer 10 School 452 Grade 2 Week 5 Day 3 

Tables 351 - 356 Observer 10 School 452 Grade 5 Week 5 Day 3 

Tables 357 - 364 Observer 10 School 452 Grade 2 Week 15 Day 5 

Tables 365 - 370 Observer 20 School 504 Grade 2 Week 4 Day 1 

Tables 371 - 378 Observer 20 School 504 Grade 2 Week 15 Day 2 

Tables 379 - 387 Observer 20 School 504 Grade 5 Week 15 Day 2 

Tables 388 - 393 Observer 30 School 433 Grade 2 Week 4 Day 4 

/ Tables 394 - 402 Observer 30 School 433 Grade 5 Week 16 Day 2 

Tables 403 - 410 Observer 40 School 440 Grade 5 Week 4 Day 4 
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Table 345 

PhaM XV - XOE evaluation projact 
Daicr I P t i vo Study - Spring 1976 

Pal lability on oanaral Content 
Obiorvar 10 Export 90 
School 452 Orada it 
waak 5 Day 3 

WDRSD 



Catagory choson Cat* 
by e*pgrt 90 * 



Cataoory choian by Obstrvar 10 
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0 


0 


0 
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♦ 
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0 
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Totalf 
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* 
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72 
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Table 346 



Phase IV • 1GE Evaluation Project 
Descriptive Study - spring 1978 

Reliability on pace 
Observer 10 Expert 90 
School 452 Grade 2 
Week L Day 3 
WDRSD 



Category chosen 
by ExP«rt 90 



Cat. 
n 



Cateqory chosen 
by Observer 10 



Marg I na 
Tota I s 



Nom | na I 
Agreement 



Marg I na i 
Agreement 



Light »s 
Extern | on 
of Kappa 
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♦ 
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1 


♦ 




I 
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4 
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♦ 
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2 


4 


3 


26 


2 


4 


31 


4 
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♦ 
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Ne 1 ther 


3 


♦ 


0 


0 


35 


4 


35 


4 
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♦ 




























♦ 
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7 


27 


38 


♦ 


72 
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category Marginal Agreement! 
Cohen's Keppel .831 
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Table 14 7 

Phase IV • IGE Evaluation Project 
Descriptive Stud/ - Spring 1978 

Reliability or Group I ng 
Observer 10 Expert 90 
School 452 Grade 2 
week 5 Day 3 
WDPSD 



Category chosen 
by Expert 90 



Xnd I v | dua I 
Pair 

Small Grouo (3-8) 

Large Group t>8) 

None of the above 



Category chosen by Observer 10 

Marginal 

j 2 3 4 5 Totals 



♦ 
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♦ 

♦ 
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4 *■ 
♦ 
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♦ 
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0 
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♦ 
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♦ 
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Agreement 
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,493 
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Table 348 

Phase IV - I<JE Evaluation Project 
Descriptive Study - spring 1978 

Re I I eb I I I tv on Mater lal s 
Observer 10 Expert 90 
School 452 Grade 2 
week 5 Day 3 
WDRSD 
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Mater | a l J Paper/penci I 
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by Expert 90 
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Marginal Totals ♦ 



28 



38 
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Table 349 

PHaM IV - tot Evaluation Project 
OtlcriPtlvt Stud/ «. spring 1976 

pal lability on uoarnar novai 
Obtarvar 10 Export 90 
School 452 Qr*d* Z 
waak s Day 3 

WDRSO 



Catagory ehotart 



Cat, 



Cataoory choaan by Obtarvar 10 



Harolnai 



light's 
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by Expart 90 


* 


1 


2 


3 
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? 


Total t 
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Agraamant 
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I 
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0 


0 


0 
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3 


♦ 0 


i 


14 


0 
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0 
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0 


0 
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0 


0 
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• soo 


1 ,000 
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6 
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,049 
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Table 350 

Phasa IV - 106 Evaluation Projact jk 
Oitcrlptlv* Study * Spring 19T0 

Riliabllltv on Intaractlont 
Obaarvar 10 Expert 90 
School 452 Of ad* 2 
waak s Oav 3 
WORSD 



o 

4S 



Program! PEL.P9S 
Oata F| las PELD3 



Catagory enoian 
by Expart 90 



Cat 9 



Taroat to Taarhar 



Taroat to studant 
Targat to Tcnr i orp 
raachar to Targat 



Studant to Targtt 
Taac*ar to Group 
0t*ar Adult to orouP 
Student to ft'our* 
HO** of tha abova 
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4 
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3 ♦ 

* ♦ 
♦ 
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♦ 
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♦ 
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♦ 
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♦ 
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0 
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0 
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0 
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0 

0 

0 

0 

0 

0 

0 

0 

0 

0 
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0 
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0 

o 
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0 

1 

9 
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0 



0 
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0 
0 
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0 
0 
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Total tt 
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0 
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T 

0 

ft 
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Table 351 

PhBM IV - IOE Evaluation Projact 
OalcrlPtlva Study • Spring 1*78 

payability on oanarai Content 
Obaarvar 10 EKPtPt «0 
SCHOOl 452 Oradt « 
wttk 5 oiy 3- 

WDASD 



Category cfco»t>n 
by t»P*ft $0 



nalt 

Trant i t ion 
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1 2 3 * « 6 
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Table 352 

Phase IV - IGE Evaluation Project 
Descriptive Study - spring 1978 

Pal labl I I ty on Pace 
Observer 10 Expert 90 
School 452 Grade 5 
Week 5 Day 3 
WDRSD 



4> 
O 



Cateaory chosen 
by Observer 10 



Lloht»s 



Category chosen 


Cat, 










MerQ 1 na 1 




Nom I na I 


Marg I na I 


Extent | on 


by Expert 90 


# 




1 


2 


3 


Totals 




Agreement 


Agreement 


o* Kappa 






♦ 








♦ 


♦ 








Set t 


1 


♦ 


8 


6 


0 
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♦ 


♦ 








Other 


2 


♦ 


5 


43 


0 


+ 48 


♦ 
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,960 
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♦ 


♦ 








Ne 1 ther 


3 


♦ 


1 


0 


3 
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♦ 
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• 
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*9 
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Table 353 

Phase IV - IGE Evaluation Project 
Descriptive Study • Soring 1978 



pel labl I 1 1 y 
Observer 10 
School 452 
week 5 



on Grouping 
Expert 90 
Grade 5 
Day 3 



WDRSD 



Category chosen 
by Expert 90 



Category chosen by Observer 10 



Cat, 
n 



Marg } na I 
Total s 



Nom | nal 
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Marg I na | 
Agreement 



Light's 
Extens I on 
of Kappa 
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1 


♦ 
♦ 
♦ 


8 


0 


0 


2 


0 


♦ 

♦ 10 
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♦ 


0 


0 


0 


0 


0 


♦ 0 
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3 


♦ 


0 


0 


0 


0 


0 


♦ 0 
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4 


1 


0 


47 


0 
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5 


♦ 


1 


0 


0 


0 


3 
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♦ 


13 


1 


0 


49 


3 
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Table 354 

Phase !V - IGE Evaluation Project 
Descriptive Study - SPrlnQ 1978 

Pel lebll I ty on Mater I a I s 
Observer 10 Expert 90 
'School 452 Grade 5 ' 
Week 5 Day 3 
WDPSD 



4> 
O 



Material t Paper/pencil 



Cateqory chosen 
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Category chosen 
by Expert 90 
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+ 
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♦ 
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♦ 
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Tota I s ♦ 
♦ 



47 
1 
1 

49 



0 
1<» 

0 

14 



0 
0 
3 



Mar9 Una I 
Tota I s 



+ 
♦ 
♦ 
♦ 

♦ 

♦ 
♦ 



47 
15 
4 

66 



♦ 
♦ 

♦ 

♦ 

♦ 
♦ 



Norn | na I 
Agreement 



1*000 
.933 



,750 



UQht»s 
Marginal Extension 
Agreement of Kappa 



,959 1.000 
,933 ,915 
,750 .738 



••-\ 



Mb 



Category Marginal Agreement! .881 
CoHen»i Keppal .928 



9 

ERIC 



Tabl'e"355 

ft , al , 1V ~ iGE Evaluation Projact 
TmcM^IvI Study -Spring lf« 

v Ml lability on ttarnar Movaa 
r obiarver 10 Expart 90 
school 452 Grnda a 
5 Day 3 
WORSO 



waa* 

■ ft* 



Category chotan 
by txP^rt f i0 



tnoagtd - wr I t tan 
Enoaoad w oral 
tngagad - covart 



Cat . 
a 



Catagory choaan by ObMrvar 10 



Marginal 
Total* 



1 ♦ 

4 

2 ♦ 
♦ 

3 ♦ 



0 



IT 



l - 2 3 4 5 6 T 

«..♦*..♦*..♦♦ 

10 
0 

19 
3 
21 
9 
4 

66 



0 



Z 
0 



13 
1 



\ 
6 



hot angagad - Intarlm 4 ♦ 
Not angagad - waiting 5 ♦ 
Not ano#g*d -oM tat* 6 ♦ 
Nona o# tha abova ▼ ♦ 



, ! 4 r 35 3 10 11 3 
H , r , * g | Totali ♦ 11 



1 



♦ 

♦ 

♦ 

♦ 

♦ 

♦ 

♦ 

♦ 

♦ 

♦ 

♦ 

♦ 

♦' 
♦ 



Nominal MarglnrfK 
*gr*«mant Agreamant 



,400 

• 695 
,333 

• 23a 

• 667 
,750 



,400 

,543 
1,000 
,4tb 
,816 
«750 



category Marginal Agraamanti 
Cohanta Kapptl i*0* 



fa 70 



• Table 356 

Phii* IV ~ IOE Eveluetlon Project 
Descriptive Study • spring 1976 

Reliability on interaction! 
Observer 10 Expert 90 
School 452 Grade 5 
Weak 8 Day 3 
WORSO 



Datet 06»2S*79 
Programi RELP** 
Data F I lal REL.03. 



Category cHoien 
by Expart 90 



Cat* 



Target to Teacher 
Target to Other Adult 
Target to Student 
Target, to Tchr fi, Orp 
Teacher to Target 
Other Adui t to Yarget 
Student to Target 
Taaeher to oroup 
Otherf Adul t to Group 
Student to oroup 
hone 0* tha above 



t;7 1 



9 

ERIC 



Category choten by Observer 10 

3 4 S 6 T B 9 10 



11 



Herginal 

tetali, 

♦♦♦♦♦♦♦♦ 



Now I nal 
Agreement 



Ulg*t»i 
Marginal EKtam |on 
Agrea^nt e* Keppe 



Ottaoory Merpine! Agreewtntt #300 
Ophen»« Keppal #3B9 



♦♦♦♦♦♦♦♦ 



1 


♦ 
♦ 


0 


0 


0 


0 


0 


0 


0 , 


0 


0 


0 


0 ♦ 


0 ♦ 




2 


♦ 

♦ ' 


0 


0 


0 


0 


0 , 


0 


0 


0 


0 


0 


o ♦ 


0 ♦ 




3 


* 
♦ 


0 


0 


0 


0 


0 


0 


0 


0 


0 


o 


1 » ♦ 


1 ♦ 


OtOOO 


A 


♦ 
♦ 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 ♦ 


0/ ♦ 




5 


♦ 

4 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 ♦ 


0 ♦ 


• ft> 


6 


♦ 


0 


0 


0 


0 


0 


0 


0 


0 


0 1 


0 


0 ♦ 


0 ♦ 




7 




0 


0 


0 


0 


0 


0 


0 


0 


0 


0 


i ♦ 


1 ♦ 


OtOOO 


6 


♦ 
♦ 


0 


0 


0 


0 


0 


0 


0 


10 


0 


A 


i» ♦ 


89 ♦ 




9 


♦ 


0 


0 


* 0 


0 


0 


0 


0 


0 


0 


/o 


0 * 


0 ♦ 




10 


♦ 


0 


0 


0 


0 


0 


0 


0 




0 


3 


1 « 


> A ♦ 


,790 


11 




0 


0 


0 


0 


0 


0 


0 ' 


>; 


0 


2 


26 i 


► 31 ♦ 


,903 




♦ 




















ill 




0 


0 


0 


0 


0 


0 


0 


11 


0 


9 


46 t 


► 66 ♦ 





o^ooo. 



0,000 



0,000. 
,379 

,674 



0,000 
,2U 

.711 
,661 



K7; 



Table 357 

PhaM IV - IOE Evaluation Project 
DaicrlPtlva Study • Spring l9jP« 

) \ 
Rtl lability on Qanfral Contant\ 
Obiarvar 10 $*Part 90 
School *52 Grada 2 4 
v ' waak 15 Day 5 
WDRSD 



Catagor? chosan 
by Expart 90 



Halt 

Trans 1 1 1 On 

PanagGman't* 

IraaU 

fsonacadam | c 
Othap Acadamlc 
Nona o# tha Abova 



Catt 



1 

2 

3 

4 

5 

6 

7 



Marginal Totait 



Catagory choaan by obtarvar 10 



0 

i 

p 

0 
0 
0 

3 7 



7 
0 
0 
0 
0 
0 



' 3 



0 

( 

3 

0 

0 

0 

0 



) 



Marginal 
Totals 



Nominal 
Agraamant 



Marginal 
AQraamant 



12 



♦ 



72 



♦♦♦♦♦♦ ♦♦♦♦♦♦♦♦ 



LIOHt»l 
Extans Ion 
of KePPa 



0 


0 


0 


0 


♦ 
♦ 


t ♦ 


1.000 


.667 


4.00.0 


0 


0 


0 


0 




T ♦ 


1 ,000 


1 ,000 


1,000 


0 


0 


0 


0 




J ♦ 


,667 


,667 


,657 


3 


0 


? 


0 


♦ 


3 ♦ 


1 ,000 


1 ,000 


1,000 
-»««••' 


0 


0 


0 


0 


♦ 


0 ♦ 


.aja*aa«p«a 






0 


0 


12 


0 


♦ 
♦ 


12 ♦ 


1,000 * 


1,000 


1,000 


0 


0 


0 




♦ 


*8 ♦ 


1,000 


1,000 


. 1,000 



Cataoory Marginal Aoraamantl ,800 
Cohan'f Kappal ,*76 

* 



673 



,y , Table 258 



Phase 'IV - IGE Evaluation Project 
Descriptive Study - Spring 1978 



. pel lab I I I ty on Pace 
Observer 10 Expert 90 
School 452 Grade 2; 
Week 15 Day 5 
* WDRSD 



\ 



/ 



Category chofen, 
by Expert 90 



Self 
Other 
Neither 



Cat, 



1 ♦ 
♦ 

2 ♦ 



Category chosen 
by Observer 10 



1 



3 ♦ 
♦ 

♦ 

Marg I na I Total s ♦ 



Marg I na I 
Totals 



Nom I na | 
Agreement 



Marginal .. 
Agreement 



L Iflht • s 
Extens | on 
of Kappa 



13 


1 


0 


♦ 


14 


♦ 


• 929 


.875 


,908 


3 


26 


. o 




31 




#903 


,935 


. .838 


0 


0 


4 27 




17 


> 


1,000 


1,000 


1,,000 






1 






























16 


29 


27 




78 


♦ 









1\ 



category Marginal Agreement! ,937 
Coh«n'i Kappat .91* 



67 o 



■ * 



Table. 359 

Phase IV », IOE Evaluation project 
Descriptive Study-Spring 1978 

■Reliability on oroup'in/ 
Observer- 10 • Expert 90 
School 452 Grade 2 

week 15 Day 5 > 
WORSO 



V. 



•Category chosen 
by Expert 90 



Category chosen by Observer 10 



Cat, 



Mar g | na 
'Total s 



Norn I na I 
Agreement 



Marg |na | 
"Agreement 



Light's 
Extens I on 
of Kappa 



















♦ 




♦ 








Indl v 1 dua 1 


1 




7 


0 


1 


0 


0 




8 ' 


4 


. .8*5 ' 

V* 


• 615 


. ,847 


'Pair 0 " 


2 


♦ 


5 


5 


0 


0 


0 




10 


♦ 


t500 


,900 


,429 






C 












♦ 




♦ 








$ma| | Group (3*8) 


3 


1 


4 


1 


0 


0 




6 


♦ 


.167 


.500 


,130 






♦ 












♦ 




♦ 


.952 V ' 






larg© orouP J>Q) 


4 


♦ 


0 


0 


1 


20 


0 


♦ 


21 


♦ ' 


.952 


' ,934 


None of the abov.e 


5 




0 


0 


0 


0 


27 


■f 
♦ 


27 


♦ 


1.000 


1.000 


1,000 
































♦ 








4 








♦ 








Marginal Tota 


s 


♦ 


13 


9 


3 


20 


27 " 




72 











/ 



•b 



676 



9 

ERIC 



Category " Marg I na | Agreementl ,794 
Cohen »s Kappa I ,774 



G77 



u> 



Table 36U 

Phase TV - I r 'E Evaluation Project 
Descrlotlve Study - sop I no 1978 

Re liability on. Mater I a I s 
Observer 10 Expert 90 
School 453 Grade 2 
Week 15 ^\Day 5 

'worst 




4> 
H 



Material I Paper/penc I I 



Category chosen 
by Expert 90 



used 

Not used 
\^ther 



Maro I na 



Cat e 



Category chosen 
by Observer 10 



1 



Marginal Nominal Marginal 
Totals Agreement Agreement 




,958 
1.000 
1,000 



,958 
,955 
1,000 



L I ght » s 
Extens ion 
of Kappa 



,939 
1,000 
1,000 



\ 



G73 



Category Marginal Agreementl 
• -Cohen's Kappal ,979 



• 9T1 



■+ 

/ 



• / Table 361 

phase IV * IOE Evaluation Project 
Descriptive Study - Spr Ins 1978 v 

Reliability o(h ^ater ! a I s , 
Observer 10 \ Expert 90 
School 452 Orade. 2 
Week 15 Day- 5 
WDRSD 



1 v 



Category chosen' 
. by Expert 90 



used 



Not used 



Other 



Cat* 



\ 



Materia)! Printed material 



s 



Categqryv chosen 
by Obsef van lo »■ 



■ ♦ 

1 ♦ 
♦ 

2. ♦ 
3 

♦ 



.Marginal .Nominal' Marginal 



L lohytt s 
Extens I on 



1 

0 
0 
0 



/2 

V 2 
43 
0 



Total's Agreement Agreement of Kappa 



A* ♦♦♦♦ 

♦ ' 1 i. ♦ 

♦ 



, o 

0 
27 



♦ 
♦ 



2 
43 
27 



♦ . 

. * 

♦ 
♦ 



1.000, 

l.ood 



4 n 



C ,000 
• 956 
1#000 



0,000 
l.OOO 
1,000 



Marg I n a » Tota I i ♦ 



J 



45 27 



4 
♦ 



72 



♦ 



6b0 



category Marginal Agreement! #652 



Cohen»s Kappa! »943 



/ 



681 



Table 362 

Phase XV - I<5E £ va I uat I on Project 
Descriptive Study - SPrlnO 1978 

* Re I lab I II t v on Materials 
Observer in Expert 90 
School. 452 Grade 2 
Week 15 'Day 5 
. WDRSD 



Mater I a I I Can ' t Te I I 



i — ■ 



Category chosen 
by Expert 90 



Used 

Not used 
Other 



Oat. 



Cateaory chosen 
by Observer 10 



1 



Marg I na I 
Tota I s 



Nom | na I 
Agreement 



Mar g | na I 
Agreement 



Marg I na I 



Totals ♦ 
♦ 



45 



27 



4- 
♦ 
♦ 



72 



♦ 
♦ 



Category Marginal Agreementl .659 
Cohen's P<appal .971 



Light ts 
EXtens | on 
of Kappa 













♦ 




♦ 








1 




0 


. 1 


0 




1 




0.000 


0..Q00 


0,000 












4- 




♦ 








2 




0 


44 


0 




44 


♦ 


1 .000 


.978 


1.000 












♦ 




♦ 








3 




0 


0 


27 




27 


♦ 


1.000 


1 .000 


1.000 












♦ 













r. 



7 Table 363 

Phase IV - IGE Evaluation Project 
Descriptive Study- Spring .975 

Reliability on Learner Move* 
Obeerver 10 E*pert 90 
School *58 Grade 8 
Week*U5 Day S 
* W0R9D 



Category choien 
by Expert 90 



Engaged • wr | ttan 
Engaged • oral 
Engaged " - covart 



Cat i 
n 



♦ 

l ♦ 

3 ♦ 



Category choeen>y Observer^y^ 



1 



19 



hot engaged - interim * ♦ 

hot angaged • waiting 5 ♦ 

Not engaged «oM task 6 ♦ 0 

Nona of tha abova ▼ ♦ 0 

♦ 

Marginal Totalt ♦ 7 
♦ 



a? 



Marginal 
Totals 



2* 



at 



Nominal Marginal EKtanalon 
Aoraamant agraamant o* KaPPa 




,685 
,667 
.798 
1.0,00 
,*00 
1,000 
1,000 



,875 
1,000 
,958 

1,000 

4 

,400 
,375 
1,000 



.585 
,658 
,69* 
1,000 
,383 
1,000 
1,000 



Cataoorv Marginal aoraamant I ,801 
Cohan't Kaopai * ,773 



er|c 



684 



Table 364 

phete IV • IOC Evaluation Project 
Descriptive Study • spring, 19TB 



4> 
M 
00 



Dettl 06*25-79 ' 
Program! ' RELP9S 
Data rilt! .REUD3 



Reliability on Interactions 
Observer 10 



Expert 90 
School *52 Graded 
weak 18 Day * 
WPftSO 



Category c^otan 
by Expert 90 



Cat. 











Taroat to Teacher 


1 


♦ 


0 


Taroat to Other Adult 


8 




0 


Taroat to Student 


3 


♦ 


0 

f 


Target to Tehr * Orp 


k 


♦ 
♦ 


0 


Teacher to Target 


5 


♦ 


0 


Other Adult to Target 


6 




0 


Student to Taroat 


7 




0 


Teacher to Oroup 


6 




0 


Other Adul t to oroup 


9 


♦ 


0 


student to oroup 


10 




0 


None of the above 


11 




0 














♦ 





1 3 

0 
0 
0 
0 
0 
0 
0 
0 
0 
0 



Marginal Totalt ♦ 



685 

4 



0 

ERIC 



Category 


choaen 


by 


Observer 


10 






Marginal 


NOm | HO 1 


K 

Mara ( ni 1 




*> 


5 


6 


T 


e 


9 


10 


•11 


Total! 


AOreement 


Agreement 






















\ 


1 


0 


0 


0 ' 


0 


0 


0 


0 


► 1 ♦ 


0*000 


0*000 


0 


0 


1 


0 


0 


* 0 


0 


0 




0t000 


ItOOO 


0 


0 


0 


0 


0 


0 


0 


" 0 


0 


\\ : -\ 




o t ooo 


0- 


0 


0 


0 


0 


f 

0 


0 


0 


0 






0*000 


0 


0 


1 


0 


0 


1 


0 


0 


0 




*% 


*600 


0 


0 


0 


0 


0 


0 


0 


0 


0 


♦ ' 0 « 




0*000 


0 


0 


0 


0 


0 


0 


0 


0 


0 


♦ 0 " < 




0*000 


0 


0 


0 


0 


0 


11 


0 




1 


♦ 1* 


> ,T0» 


*«75 


0 


0 


0 


0 


0 


0 


0 




0 


♦ . o < 






1 


0 


0 


0 


0 


1 


0 




0 


♦ t 


► 0«000 


»soo 


0 t 


0 


0 


/ 


1 


.1 


0 








¥ ,«0* 


*9*t 



ujght*t 



l 1 l* o 



♦♦♦♦♦♦♦♦ 
♦♦♦♦♦♦♦♦ 



Category Marginal Agreementl f 3§* 
Cohan's tfeppal ••11 



o«ooo 

• ♦OH 



.*9J 

«.0U 
• 690 



6 bo 



i 



Table 365 \ 

Phaf IV - IOE Evaluation -ProJM'. 
DaicrlPtlva Study - spring l«7ft 

Rat lability on oenaral Contant 
Obtarvar 20 Expart 90 
school 50* ' Orada * 

' * tfeaM • * Day l 
WDRSD 



Catagory c*o«an 
by EfcPart 90 



Cat* 



Halt 


1 ♦ 


/ 0 


0 


Trantit ion 


a ♦ 


0 


3 


fanagamant 


3 4 




0 


Braak 


A 4 


0 


<p 


Nonacadami c 


5 4 


> 0 


0 


Othar Acadamle 


a < 


► 0 


0 


Nona of tha Abova 


T < 


► 0 


0 



Marginal Total* 



catagory choaan by obaarvar 20 
18 3 « 6 7 

/ 

♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 

o o 1 o 



Marginal 
Totaii 



♦ 



Nominal Maroinal 
Agraamant Agraamant 



LlOHt»l 
Ewtani Ion 
of KaPPa 



0 
0 
0 
0 
0 

i 0 

0 3 0 



0 
0 

A 

0 
0 



0 
0 
0 
0 
0 
0 



4 (o 



0 


0 




o '• ♦* 








0 


0 


♦ 


3 ♦ 


. 1,000 


1.000 


1,000 


0 


1 


♦ t 


1 + 


**#000 


0.000 


• 0,000 


0 


0 


♦ 

♦ 


A ♦ 


1.000 


1.000 . 


1,000 


6 


0 


♦ 


0 ♦ 








0 


0 


♦ 
♦ 


0 ♦ 


• as«aj»«9aj» 


-.^•a»a»f*- 




0 


*6 


♦ 

♦ 


*6 ♦ 


1*000 


.tTt 


1,000 


▼ 

♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 








0 


AT 


▼ 
♦ 
♦ 


SA ♦ 









catagory Marginal Aoraamanti 
Cohan »• Kappa I 



687 



Table 366 

Phase IV - IGE Evaluation Project 
Oescr jpt I vtL-Study - Spring 1978 

Rel |abl I I ty on Pace 
Observer 20 Expert 90 
School 504 Grade 2 
week 4 Day 1 ' 

WDRSD * 



4> 



Category chosen 
by Expert 90 



6 b a 



self- 
other 
Ne I ther 



Category cnos«n 
by Observer 2o 



Cat • 



1 

♦ 



1 ♦ 
♦ 

2 ♦ 
♦ 

3 ♦ 
♦ 



Marginal Totals ♦ 
a ♦ 



19 



28 



\ 7 



Marg I na I 
Total s, 













14 


9 


0 


♦ 


14 








♦ 




5 


27 


0 


4 


32 








♦ 




0 


1 


7 


♦ 


e 



♦ 
♦ 



54 



♦ 



Niom I na I 
Agreement 



l'tOOO 
,844 
,875 



category Marginal Agreement! 
Cohen»s Kappal ,809 



Marginal 
Agreement 



,737 
,875 
,875 



Light is 
Extens ion 
of Kappa 



1,000 
,675 
t O 5*6 



4* 

f 



829 



i) J J 



x. 



Table 367 • 

Phase XV - IGE Evaluation Projact 
Descriptive study - Spring 1978 

• J % 
> Re I I eb I 1 1 ty on ordupina 

Observer 20 ExPeYt 90 • 

School 504 Grade 2 

week 4 Day 1 

" WORSE) 



V \ 



Category chosen 
by Expert 90 



Individual 
Pair 

Small Group <3-B) 
Large Grou^(>8) 
None of the above 



Cat. 

H 



category chosen by Observer 20 



♦ 

♦ 

1 ♦ 
♦ 

2 ♦ 

- ♦ 

3 ♦ 
♦ 

4 ♦ 
♦ 

5 ♦ 
♦ 



Marginal 
Total s 

♦♦♦♦♦♦♦♦ 



15 


0 


0 


1 


0 * 


16 


0 


q 


0 


0 


0 i 


> 0 


r 

o • 


0 


0 


' 0 


0 < 


I- 0 








r s 






3 


. l 


0 


26 


*0 - 


► 30 


0 


0 


0 


1 


7 


* a 



Marginal Totals ♦ 

♦ 




Nominal -Marginal 



Light's 
Extens I on 



Nominal • nwi »?■■—» - 

Agreement Agreement of Kappe 



♦♦♦♦♦♦♦♦ 
54 

♦♦♦♦♦♦♦♦ 



category Marginal Agreement* .674 
Cohen»s Kappal .813 



630 



,938 



,867 
,875 



,889 
0,000 

,933 
,875 



.691 



,906 

,723 
, .656 



40 



N5 



Table 368 

Phase TV - KiE Evaluation Project 
Descrlotlve Study - SPrlnO 1970 

Rol labl 1/ tv on Materials 
Observer 20 Expert 90 
v Schoo I 504 Grade 2 
week <♦ Day 1 
WDRSD 



\ 



Mater |a|j Paper/penc 



\ 



ERJC 



Categ6ry chosen 

'• by Expert 90 



\ 



Used 

Not used 
Other 



Cat, 



♦ 

1 f 

2 ♦ 
♦ 

3 ♦ 
♦ 



Marginal Totals ♦ 

♦ 



Category chosen 
by Observer 20 



1 

# 


2 


3' 


» 1 


0 


. 0 


2 


15 


0 


0 


1 


7 



Marolnli 
Total • 



29 
17 
8 



Now | na I 
Agreement 



wooo 

• 882 
" ,875 




f, r >2 



category Marginal AgraementI 
Cohen' s Kappal { t 90<* 



Marginal 
Agreement 



• 935 

• 941 

• 875 



UQht»S 
Extension 

of Kappa 



,917 



( 




/ 



Table' 369 



•Phata IV,- IOC Evaluat Ion Projaet 
OotcrlPtlva. Stgdy - SPrino 19T8 

Re M ak?|,nty on Learner Movee 

Observer 20 Expert *0 
\ School 504 - Grade 2 
wttk * Day 1 

WDRSO 



Category tHoten' 
by Expert 90i 



Cat, 
# 



1 ♦ 
♦ 

2 ♦ 



Engaged • wrl tten 

Engaged ■ oral » 

Enoagad • covert 

Net engaged * Interim * / 

♦ 

Not engaged -waiting 5 ♦ 
1 ♦ 

Not engaged -oM^tetK 6 ♦ 
None of the abpve 



0 



♦ 
T ♦ 



Category choaen by Obaervtfr 20 
1 2 3 * S 6 



e 

0 
2 
0 
0 
0 
0 



0 


0 


0 


0 


0 


0 


0, 


1 


0 


1 


0 


0 


0 ' 


9 


0 




0 


0 


0 


0 


0 


1 

• 


0 


0 


1 


10 


2 


0 


i 


0 


0 


1 


0 


0 


0 


0 


0 


1 


0 


0 


0' 


T 



'Warfll nal 
Totals 

♦♦♦♦♦♦♦♦ 



Ught«t 

Nominal Marginal Extant lorn 
Agraamant agraamant o# Kbppo t 



Marginal Total» ♦ 

♦ 



10 J 22 2 IV » t 



8 ♦ 


1*000 


,800 


1.000 


J > 


1 ♦ 

15 


50,000 
• 692 


1.000 " 

.8^1 


..oi* 
,*li 




1 ♦ 


0*000 


.500 


• .038 






.364 


.500 


.80l' 




1 ♦ 


0.000 


1.000 ♦ 


• ,0l» 


V 


6 » 


.e.TB 


,078 


,086 








* 

















Category Marginal Agreement. iT82 
Cohenta Keppet #472 



4> 

ro 
u> 



694 



Category chosen 



Cat, 

a 



♦♦♦♦♦♦♦♦♦ 
♦ 



Target to Teacher 



1 ♦ 
♦ 

Target to Other Adult 2 ♦ 

♦ 

Target to Student 



Target to Tchr I Orp 
Taacher to Target 



3 ♦ 
♦ 

k ♦ 
♦ 

5 + 
♦ 



Other Adult to target * ♦ 

♦ 

Student to Target 



7 ♦ 
♦ 

8 ♦ 

9 ♦ 
♦ 

10 ♦ 
♦ 

11 ♦ 
♦ 

♦ 

Marginal Total* ♦ 0 0 
♦ 



Teacher to Ciroup 
Other Adult to orouP 
Student to oroup 
None e* the above 



0 
0 
0 
0 
0 
0 
0 
0 
0 
0 



0 
0 
0 
0 
0 
0 
0 
0 
0 
0 
0 



Table 370 

Phase Iv - tOE Evaluation Project 
Descriptive study - spring 1978 

Reliability on intaractlons 
Obiarver 20 Expert 90 
School 50* Grade 2 
Week * Day I 
WDRSO 



Category chosen by Obsarver 20 



695 



0 


0 


0 


0 


0 


-0 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 


t> 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 


1 


0 


0 


0 



0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

c 



1 

0 

0 

o 

0 
0 
0 
8 
0 
0 
7 

10 



0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 



10 u 



0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 



0 
0 
0 
0 
0 
0 
0 
6 
0 
0 
37 

o 



Category Marginal Agreement! i*39 
Cohen's Kappai #10} 



9 

ERLC 



Datal 06*2§*T9 
Program) REIP9S 
Data F| lae REU03 



Marginal 
Total • 

+ + + * + 

1 

0 

0 
0 
0 



Ugfcttt 

Nominal Marginal Extant ion 
Agreement AQraamant Kappa 



0*000 



0,000 



0,000 



0,000 



0 

a 

0 
0 
*B 

►♦*♦♦♦♦♦♦« 
► ♦♦♦♦♦♦♦♦< 



,250 



,609 



,060 



,822 



,127 



* 



f 



^Table 371 



Phata IV - IOE Evaluation Projict 
Oa«crlPtlva Study - Spring 1978 

Reliability on Oaneral Conttnt 
Obtarver 20 Export 90 
School 504 Groda 8 
weak' 15 Oay 2 

WORSO 



getagory choten 
by CMpar t 90 



Walt 

Trantlt Ion 
Hanagamant 
iraeK 

Nonacadam I c 
Othar Academic 
None o# tha Above 



Cat v 



1 ♦ 

♦ 

2 ♦ 
♦ 

♦ 

4 ♦ 
♦ 

5 ♦ 
♦ 

6 ♦ 
♦ 

7 ♦ 
♦ 



category choaan by Obtarvar 20 

v 



1 

0 
0 
0 
0 
0 
0 



0 

4 

0 
0 
0 
0 
0 



Marolnal 
7otaia 

♦ <>♦♦♦♦♦♦ 



Nominal Marginal 
AoraemaM Agroamant 



Lights 
Cxtan*lon 
6* KaPPa 



o 

0 
5 
0 
0 

0 
0 



0 

0 

0 

0 
0 
0 
0 



♦♦♦♦♦♦♦ 

% a n ft 0 0 44 
Marginal Total! ♦ 1 * * 

* ♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 



♦♦♦♦♦♦♦♦ 



0 


0 


0 


♦ 1 


♦ 1.000 


1.000 


1.000 


0 


0 


0 


♦ 4 


♦ 1.000 


l.ooo 


*.ooo 


0 


0 


0 


♦ 5 


♦ .H°oo 


1.000 


l.ooo 


0 


0 


0 


♦ 0 




V 




0 


0 


0 


* 0 






















0 


0 


0 


♦ 0 






•* 


0 


0 


44 


♦ 44 


♦ 1.000 


1.000 


1.000 



C*t«oory M«r«in«l »0P«t««nti 1»000 
CoHtntt K«PP*I l»000 



697 



Categopy chosen 
by Expert 90 



Sel f 
Other 
Ne I ther 



„ Table 3/2 

Phase IV - IQE Evaluation Project 
Descriptive Study - spring 1978 

Rel lebl I I tv on pace 
Observer 20 Expert 90 
School SO* Orade .2 
week 15 Day 2 
WDRSO 

p\ : 

CeteqorV chosen 
by Observer 20 



4> 



Cat . 
# 



Mara I ne I 
Total s 



♦ 

1 ♦ ' U o 

2 ♦ . 1 32 
♦ 

3 ♦ 0 0 
♦ 

Marginal Totals-* 12' 32 



0 
10 



10 



♦ 
♦ 

♦ 
♦ 

4u, 

♦ 
♦ 



11 

33 
10 

5* 



Norn | na I 
Agreement 



Maro | na I 
Agreement 



1.000 
• 970 
1*000 



Light »S 
Extens I on 
of Kappa 



.917 1.000 
,970 .926 
1.000" 1.000 



Category Marginal Agreement! .962 
Cohen** Kappal .967 



9 

ERIC 



'Table 373 

Phase IV - I6E Evaluation project 
Descriptive Study - Spring 1978 



Re I l ab I I l ty on Group I ng 
Observer 20 'Expert 90 
School 50<» Grade 2 
week 15 Day 2 
WORSO 



Category chosen 
bv Expert 90 



Cat « 
u 



Category chosen by Observer 20' 



Individual 




1 


♦ 
♦ 


9 


Pair 




2 


♦ 


1 












Smalt Group 




3 


♦ 
♦ 


0 


Large Group 


<>8) 


r « 


♦ 


0 


None of the 


above 


5 


♦ 


0 








♦ 





Marginal Totals ♦ 10 

♦ 



0 

m 

X 

0 

1 

*0 



5 




0 


'0 


0 




0 


0 




1 


0 


0 


31 


0 


0 


0 


10 


0 


32 


10 



MaVg I na I 
Totals 



♦♦♦♦♦♦♦♦ 
54 

♦♦♦♦♦♦♦♦ 



Nominal 
Agreement 



Marginal 
Agreement 



Llght«s 
Extern Ion 
of Kappa 



9 4 


ltOOO 


• 900 


1..000 , 


2 * 


,500 


1,000 


• *ei 


1 i 


* o.ooo 


0,000 


0,000 


32 < 


> ,969 


1,000 


,923 


10 ■< 


► 1,000 


1,000 


1,000 



700 



category Marginal Agreement I ,780 



Cohen »s Kappa t .905 



9 

ERIC . 



701 




Table 374 



Phase IV - IGE Evaluation Project 
.Descriptive Study = spring 1978 

Ral lability on Materials 
Observer 20 Expert 90 
School 504 Grade 2 
w'eok 15 Day 2 
WORSD 



S3 
0? 



Mater | a I ? PaPer/penc j 



Category chosen 
by Expert 90 



s 



Cateaory chosen 
by Observer 20 



1 



Marol na 
Total s 







♦ 








♦ 




Used 


, 1 




12 


. 0 


0 


♦ 


12 






.♦ 








♦ 




Not used 


2 




0 


32 


0 


♦ 


32 






♦ 








♦ 




Other 


3 




G 


0 


10 




10 










































♦ 




Maro 1 na 1 


Toto I s 




12 


32 


10 


♦ 


54 














♦ 





♦ 
♦ 
♦ 

♦ 



Nominal 
Agreement 

1 #000 
ltOOO 



1.000 

9 



Marginal 
Agreement 



UO00 
UOOO 
ltOOO 



U I Oh 1 1 s 
Extens | on 
of Kappa 



1. 

1,000 
1,000 



Category Marginal Agraementr 1,000 
Cohen** Kappal 1,000 



703 



ERIC 



Table 375 v 

Phase IV - IGE EvalU-atlpn Project 
Descriptive Study - spring 1978 

' Re i |abl I I ty on Mater lal s 
Observer 20 Expert 90 
School 504 Grade 2 
Week 15 Day 2 
WDRSD 



Material I Printed material 



Category chosen 
by Expert 90 



used 

'Not used 
Other 



Cat • 



♦ 

1 ♦ 

, ♦ 

2 ♦ 

3 ♦ 
♦ 



Category chosen 
by Observer 20 



4 
0 
0 



Marginal Totals ♦ 

♦ 



0 
40 
0 

40 



0 
0 
10 

10 



Marginal 
Total s 



r i 



♦ 
♦ 
♦ 

♦ 
♦ 
♦ 



4 
40 
10 

54 



•t> 

♦ 
♦ 

+ 

♦ 



Norn | na I 
Agreement 



1.000 
l .000 
1 ,000 



Mar g I na I 
Agreement 



1 ,000 
1*000 
I .000 



Light's 

Extens I on 
of Kappa 



1,000 
1.000 
it. ooo 



\ 



704 



Category Marginal Agreement! 1.000 
Cohen«s Kappa: l.ooo 



705 



Table 376 

Phase TV - K'E Evaluation Project 
Deacr|ot,lve study - spring 19T8 ' 

< 

Rel |abl I Itv on Materials 
Observer 20 Expert 9© 
School 504 Grade 2 
week 15 Day 2 
WDRSD 



4^ 
CO 
O 



Mater I a I i qther 



Category chosen 
by Expert 90 



Used 

Not used 
Other 

Mara I na 



Catt 
u 



Category chosen t 
by Observer 20 



1 



Margl nal 
Tota I s 



Nominal' Marginal 
Agreement Agreement 




1.000 
It opo 
l.ooo 



1.000 
1.000 
1.000 



Light»s 
Extent l on 
of Kappa 



1,000 
1.000 
1.000 



700 



category Marginal Agreement! 1.000 
Cohen' S Kappal 1 .000 



707 



IERJC 



z_ 



Table 377 

Ph..t IV - IOE Evaluation ^ojact 
OaierlPtlva Study - spring 1*70 

paM ability on L«arnar Movai 
Obaarvar 20 E*P«rt 90 
school 60* or«da i 
Waak IS Day 2 

WORSO 



Catagory eho««n 
by Export 90 



Catt 



Enoagad - wrlttan 

Cnoagad - oral 

Engaged - covart 

Not angaoad • Intarim 
Not angagad - waiting 
Not angagad -of* task 
Nona o# tha abova 



1 
2 
3 

5 
6 
7 



Catagory chotan by Obaarvar 20 
y P 3 * * 6 T 



Marginal 

TQttfll 



Nominal 
Acraamant 



Marginal 
agr«tw«nt 



K«rg|nal Total* 



2 

0 

1 

0 

0 

0 

0 



0 

1 

0 
0 
0 
0 
0 



0 

1 

13 
2 
1 
1 

0 



0 
0 
0 

* 

0 
0 
0 



0 
0 

1 

0 
7 
2 
0 



0 

0 

3 
1 
0 
H 
0 




ug*t»« 
Extant I on 
0 # K«PPa 



0 


♦ 


2 ♦ 


1,000 


• 667 


leOOO 


0 


♦ 
♦ 


2 ♦ 


• 500 


• 500 




0 


♦ 
♦ 


18 * 


• 722 


1,000 


• 503 


0 


♦ 
♦ 


7 i 


► ♦571 


• 571 


a 537 


0 


♦ 
♦ 


0 < 


► t S75 


• 000 


• 6*7 


0 


♦ 
♦ 


7 


t *571 


• 07B 


t *97 


10 


♦ 
♦ 


10 


♦ 1*000 


1*000 


1*000 



C«t«oory Marginal »orotm«nti tTTJ 
Cehtn*» k«pp«I t*«T 



ERIC 



708 



1 



Table 378 

Phete Iv - IOE Evaluation Project 
Descriptive Study - Spring 1970 

Reliability on Interaction! 
Observer 80 expert 90 
School *0M Grade 2 
weak IS Oav 2 

tfORSO 



Category eho!en Cat* 
by Expert 90 # 



Category choten by Observer 20 
3 * 5 6 7 S 9 



Marginal Yotali 



IB 



7 1) J 



10 11 



Target to Teacher 


1 ♦ 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 ' 


0 


Target to Other Adult 


2 ♦ 


0 


0 


0 


0 


0 


0 


0 


0 


0 


e 


0 


Terget to Student 


3 «• 


0 


0 


I 


0 


0 


0 


0 


0 


0 


0 


0 


Target to Tchr l Grp 


A ♦ 


0 


0 


0 


0 


0 


0 


0 


1 


0 


0 


0 


Teacher to Target 


5 ♦ 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 


Other Adult to rer get 


6 ♦ 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 


Student to Target 


T ♦ 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 


Teacher to oroup 


6 ♦ 


0 


0 


0 


0 


0 


0 


0 


10 


0 


c\ 


3 


Other Adult to Group 


9 ♦ 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 


Student to oroup 


10 ♦ 


0 


0 


0 


0 


0 


0 


0 


e 


0 


0 


0 


None o# the above 


11 ♦ 


0 


0 


0 


0 


0 


0 


0 


* 


0 


0 


30 



30 



category Marginal Agreement! #710 
Cohen's Kappei #tftl 



Marginal 
Totals 

♦♦♦♦♦♦♦♦ 

0 

0 

1 
1 

0 

0 

0 
13 

0 

0 
39 

♦♦♦♦♦♦♦♦ 
•* 



Date! 0t«2ft»79 
Prioremi ACUP9S 
Off* Flltl RELDft 



Ulflht*! 

Nominal Marginal extent | oh 
Agreement Agreement of Kappa 



1,000 1,000 1,000 

0,000 0,000 0,000 



,7(9 



,667 



peso 



• 997 



,97% 



,*S+ 



7i0 



I 



0 

ERIC 



Table 379 

Phaia IV « 1GE Evaluation Projact 
OaicriPtlva Study - Spring HT8 

pal lability on oanarai Contant 
obtarvar 20 Export 90 
School 50V Grada 5 
Waak IS Day a 
MDRSD 



Category chotan Cat» 
by Export 90 # 



Catagopy chotart by obtarvar So 
1 2 3 * 5 6 7 



Marginal 
Total* 



Marginal rotata 



♦ 



Mai t 


1 ♦ 


0 


0 


•< 0 


0 


0 


0 


0 


Tram 1 1 Ion 


2 ♦ 


0 


0 


0 


0 


0 


0 


0 


Managamant 


3 ♦ 


0 


0 


0 


0 


0 


0 


0 


•raak 


4 ♦ 


0 


0 


0 


22 


0 


0 


0 


Nonacadami c 


» ♦ 


0 


0 


0 


0 


0 


0 


0 


Othar Acadamic 


6 ♦ 


P 


0 


0 


0 


0 


0 


,1 


Nona of tha Abova 


T ♦ 


0 


0 


0 


0 


0 


0 


11* 



22 



116 



♦ 
♦ 
♦ 
♦ 
♦ 
♦ 
♦ 
♦ 

♦ 
♦ 

♦ 
♦ 

♦ 
♦ 

♦ 



0 
0 

f) 

0 
< 0 

116 

131 



Nominal Marginal 
Agratmant Agraamant 



♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 



l«000 



upoo 



Lights 
extanilon 
Of MPPa 



1.000 



uooo 



1,000 



1*000 



Category Marginal aoraamanti 1*000 
COhtmo Kappa I 1*000 



9 

ERIC 



711 



/ 



.A 



Table 380 

Phase TV - IGE Evaluation Project 
Descriptive Study - spring 19Y8 



Pe! labl I I tv on pace 
Observer 80 Expert 90 
School 504 Grade 5 
Week 15 Day 2 
WORSO 



// 



ERIC 



Category chosen 
by Expert 90 



Cat t 
u 



Category chosen 
by Observer ^0 



Marg Inel 
Tota I s 



Nom | na I 
Agreement 



Marginal 
Agreement 



L I o>t * s 
Extens Ion 
of Kappa 



Se I f 
Other 
Nel ther 



V 



f 



♦ 

1 ♦ 88 

2 ♦ ' 5 
♦ 

3 ♦ 0 

++++++++++ 



Marginal Total* ♦ 

t 



93 



0 
0 
45 

45 



♦ 

♦ 

♦ 
♦ 
♦ 



86 
5 
45 

138 



♦ 

♦ 

♦ 
♦ 

♦ 
+ 
♦ 



1.000 
0*000 
1.000 



,946 It 000 

0t000 0*000 
1*000 1*000 



7 1 ') 
/I/O 



category Marginal Agreement! *649 
Cohen's Kappei .922 



v r 



713 



Table 381 



Phase IV - 1GE Evaluation Project 
Descriptive study - Spring 1978 



pel labl I Ity on Grouping 
Observer 20 Expert 90 
School 504 6ra.de 5 
week 15 Day 2 
WDRSCr 



Category chosen 
by Expert 90 



Category chos*n by Observer 20 



Cat. 
u 



1 



2 



Marg I na I 
Total s 



Noffl I na I 
Agreement 



Marginal 
Agreement 



Llg*t»8 
Extens I oh 

of Kappa 



Ind I v I dua I 
Pair 

Smal | Group (3«8) 
Lerge Group OS) 
None of the above 



1 

2 
3 
4 
5 



♦ 
♦ 
♦ 
♦ 

♦ 
♦ 



80 
1 
0 
0 
0 



2 
9 
0 
0 
0 



0 
0 
I 
0 
0 



0 
0 
0 
0 



0 
0 
0 
0 



Marginal Totals ♦ 

♦ 



81 11 



0 45 

.J, 



4 

0 45 



♦ 
♦ 

♦ 



82 
10 
1 
0 
45 

136 



♦ 

♦ 
♦ 



• 976 

• 900 
i #000 

1 tOOO 



• 988 

• 909 
1 1 000 

1*000 



#941 
• 691 
1,000 

1 9 000 



9 

ERIC 



714 



Category Marginal Agreementl »9T4 
Cohen's Kappat «960 



715 



u> 



i 



' Table 382 

Phase IV - IGE Evaluation Project 
Descrlotlve Study - spring 1978 

Re I | ab | I I ty on Materials 
Observer 20 Expert 90 
School 904 Grade S 
,week 15 Day 2 
WORSO 



V. 



Mater i ail Paper/per\c 1 1 



Category chosen 
by Expert 90 



Cat, 
u 



Category chosen 
by Observer 20 



Marginal Nominal. 
Totals Agreement 



Lloht«s 
Mergina! Extension 
Agreement of Kappa 



Used 

Not used 
Other 



































1 




37 


5 


0 




42 






♦ 








4 










10 




0 




51 














♦ 




♦ 


3 




0 


0 


45 




45 














♦ 







• 881 
.80* 
1.000 



• 894 

• 902 
1.000 



• 819 

.7<?6 
1.000 



Marginal Totals ♦ 47 

♦ 



46 



45 



♦ 



138 



♦ 
♦ 
♦ 



716 



category Marginal Agreement! .932 
Cohen's Kappat .£37 



717 



9 

ERIC 



V 



/ 



fable 383 

Phase IV - TGE Evaluation Project 
DeicriPtlve study • Spring 1978 

Reliability on Mater I a I s ~* 
Observer 20 Expert 90 
School 504 Grade 5 
Week IS Day 2 

W0I1SD 



Material) Printed material 



Category chosen 
by Expert 90 



Cat, 

H 



Cateqory chosen 
by Observer 20 



1 



Marg I na I 
Total s 



Nominal 
Agreement 



Margi nel 
Agreement 



Llflhtts 
EXtens | on 
of Kappa 







♦ 








4 








Used 


1 


♦ 


60 


11 


0 




71 




• 845 






♦ 










i 






Not used 


2 


♦ 


2 


20 


0 


w 


22 




• 909 






















Other 


3 


♦ 


0 


0 


45 


♦ 


45 




ltOOO 


























♦ 
















Marginal 


Total s 




62 


31 


45 




138 










*> 












♦ 





.873 
.710 
1.000 



• T19 
,883 
1,000 



718 



Category Marginal Agreement! ,861 

t 

Cohan's Kappa) ,850 



719 



ERIC 



Table 384 

Pha.se IV - XGE Evaluation Project 
Descriptive Study - SprlnO 1978 

Re I | ab I I | tv on Mater I a Is 
Observer 20 Expert 90 
School 504 Grade S 
Week 15 Day 2 
WDRSD 



00 



Material l Other 



Category chosen 
by Expert 90 



Cat, 
u 



Category chosen 
by Observer 20 



1 2 3 



Marginal Nominal Meroinal 
Totals Agreement Agreement 







■f 












♦ 






Used 


1 


♦ 




1 


0 




1 




o* 


000 






♦ 


















Not used 


2 


♦ 


0 


92 


0 




92 


♦ 




000 














♦ 










Other 


3 




0 


0 


45 




45 




U 


000 














4 











0.000 
.989 

H'000 



Llghtf s 
Extens i on 
of Kappa 



0.000 
1.000 
1.000 



, Marg I na I Tota | s ♦ 

♦ 



93 



45 



♦ 

♦ 



138 



♦ 
♦ 
♦ 



7-0 



Category Marginal Agreements .663 
Cohen's Kappal .984 



721 



Table 385 



Phase IV - IftE Evaluation Project 
Descriptive Study - spring 1978 

Rel |ab| I | ty on Materials 
Observer 20 Expert 9o 
School 504 Grade 5 
Week 15 Day 2 

WDRSD 



Material t Can' t Tol I 



Category chosen 
by Expert 90 



Used 

Not used 
Other 



Category chosen 
by Observer 20 
Cat. Marginal 
# 12 3 Totals 



Marg| nal 



1 ♦ 

2 ♦ 
J ♦ 



Totals ♦ 



1 

92 
0 

93 



0 
0 
45 

45 



♦ 

♦ 

♦ 



1 ♦ 

♦ 

92 ♦ 
♦ 

45 ♦ 

138 + 
♦ 



Nom | na I 
Agreement 



0,000 
1,000 
1.000 



Marg I na I 
Agreement 



Llghtis 
Extens I on 
of Kappa 



O.ooo 0,000 
.989 1,000 
1,000 1,000 



722 



Category Marginal Agreement! *663 

Coh«n»8 Kappal ,984 723 



ERJC 



" Table 386 

Phase IV - IOE Evaluation Project 
OeecrlPtlvo Study • Soring 1975 

Ren ability on learner Movot 

Observer 20 Expert 90 

School 504 Orede 5 

week 15 Day 2 

W0RS0 



4> 
O 



Category choien 
by Expert 90 



Engaged - written 
Engaged • oral 
Engaged • covert 



Cat* 
* 



Category chosan by observer 20 
12 3*567 



Merolnal 

Total • 



♦ 

1 ♦ 

♦ 

2 ♦ 

3 ♦ 
♦ 

Not engaged «■ interim 4 ♦ 

♦ 

Not engaged waiting 5 ♦ 

Not engaged «oM tea* 6 ♦ 

♦ 

None * the ebove 



♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 

♦ 

♦ 



7 ♦ 
♦ 

♦♦♦♦♦♦♦♦♦♦ 
♦ 

Marginal Totals ♦ 22 3 



1* 


0 


1 


0 


0 


0 


0 


0 


3 


3 


43 


3 


3 


0 


0 


7 


0 


0 


0 


0 


2 


0 


0 


1 


0 


0 


0 o 


0 



44 



11 



0 
0 
0 
0 
0 
0 
0 



0 
0 
0 
0 
0 
13 
0 



0 
0 
0 
0 
0 
0 
45 



13 45 



15 
0 
52 
10 
0 
16 
45 

135 



♦ 
♦ 

♦ 
♦ 

♦ 
♦ 

♦ 
♦ 
♦ 

♦ 
♦ 



Nominal 
Agreement 



.933 



• 627 

• 700 

• 513 
ltOOO 



Marginal 
Agreement 



• 602 
0*000 

• 644 
,909 



LlOht'i 
Extent Ion 
of KePPe 



f 



• 613 
ItOOO 



• 921 

• 746 

• 674 

• 793 
1*000 



Category Marginal Agreement! *706 
Cohen u Keppei «643 



724 



Table 387 

Phftto IV - I6C Evaluation Project 
Deicrlp\lv* Study ~ spring 1978 

Reliability on interactions 
Observer 20 expert 90 
School 50* Grade S 
weak 18 Day 2 
NDRSO 



/ Oatal ^•26«T9 
/ Program! REIP9S 

Data M Iti REID3 



Category chosen Cet f 
by r«P«rt 90 * 



Marginal Totals 



Category chosen by Observer 20 
3* 5 6 7 69 



10 U 
♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦ 



Xarget to Teacher 


1 ♦ 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 


Target to Other Adult 


2 ♦ 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 


Target to Student 


3 ♦ 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 


Target to Tehr i Orp 


A ♦ 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 


Teacher to Target 


5 ♦ 


1 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 


Other Adult to Target 


* ♦ 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 


Student to Target 


T ♦ 


0 


0 


1 


0 


0 


0 


0 


0 


0 


0 


1 


Teacher to orogp 


e 4 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 


Other Adult to oroup 


9 ♦ 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 


Student to oroup 


10 ♦ 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 


0 


None o# the above 


i 

11 ♦ 


0 


0 


1 


0 


0 


0 


0 


0 


0 


0 





1 



138 



Marginal 
Total a 

♦♦«-♦♦♦♦♦ 

0 

0 

0 

0 

\ 

0 

t 

0 
0 
0 

135 
♦♦♦♦♦♦♦♦ 
131 



Nominal Marginal EKtantion 
agraamant agraamant of Kappa 



0,000 



0,000 



,t»3 



0,000 



0.000 



0,000 



0,000 



1,000 



0,000 



0,000 



,659 



or 



Cetegory Marginal Agreement! .200 
Cohen*! Kappas #32* 



7 2 (J 



4> 



Cataoory ehoaan 
bv EwPtrt BO 



Halt 

Transition 
Nanagamant 
Brank 

Nonacad^hc 
Othar Acadamlc 
Nona o# tha Abova 



Catt 
a 



1 ♦ 

2 4 

♦ 

3 ♦ 

♦ 

5 ♦ 

6 ♦ 

7 ♦ 



Marginal TotaH ♦ 



Table 388 

Ph.ia IV - IOE Evaluation Proj.ct 
DeicrlPtlvo Study - Sprinc 1«tb 

Reliability on ©anarai Content 
, Obiarver 30 E*P«rt 80 
School *33 Orodo 2 

Meek i> Day * 

OMP 



Category chotan by Observer 30 
! 8 3 * 5 6 7 

-V 

oooo 

0 



0 
0 
0 
0 
0 
0 



»*ar« I na I Nominal Marginal 
Total! Aoraamant A«raawant o# KaP^a 



t 
o 
o 

0 
0 
1 



0 
0 
0 
0 
0 
0 



0 
0 
0 
0 
0 



0 
0 
0 
0 
0 
0 
0 



0 
0 
0 
0 
0 
0 
0 



0 
0 
0 
0 
0 
0 
SI 



♦ 
♦ 
♦ 
♦ 
♦ 
♦ 
♦ 

+ 
♦ 



0 

a 

0 
0 
0 

0 



ai 



so 



♦ 

♦ i#ooo 

♦ ••••• 

♦ 
♦ 
♦ 
♦ 

♦ .Ml 
♦ 



i*67 



1,000 



,«0l 



cataoory Marginal Agreementi .8»* 
Oohtmi Kawat »Tfl 



727 



1, 



Table 389 

Phase IV - IGE Evafuatlon Project 
Descriptive Study « spring 1978 

Re l | ab I | | ty on Pace 
Observer 30 Expert 80 
School 433 Grade 2 
Week 4 Day 4 
OMP 



Category chosen 
by Expert 80 



self 



Other 



Nel ther 



Cat • 



1 



♦ 
♦ 
♦ 

2 ♦ 
♦ 

3 ♦ 
♦ 



Category chosen 
by Observer 30 



1 



Margi na I 
Total t 

♦ ♦♦♦♦ 



Nominal 
Agreement 



Light »s 
Marginal Extension 
Agreement o# Kappa 



1 






♦ 




♦ 








0 


0 


♦ 


1 


♦ 


1*000 


.043 


1.000 








♦ 




♦ 






22 


28 


1 




51 




• 549 


• 549 


.063 








♦ 




♦ 






0 


0 


2 


♦ 


2 


♦ 


1*000 


.667 


1.000 












♦ 







Marginal Totals ♦ 



23 



28 



♦ 



54 



♦ 
♦ 
♦ 



72b 



ERIC 



category Marginal Agreement) ,420 
Coh«n»s Kappa I ,149 



729 



w 



Table 330 

Phase Iv - I6E Evaluation Project 
Descriptive Study • Spring 1978 

Re I lab I l 1 1 y on Grouping 
Observer 30 Expert 80 
School 433 Grade 2 
Week 4 Day « 
OMP 



4> 



Category chosen 
by'expert 80 



Category chosen by Observer 30 



Cat, 
ft 



5 



Marg | na I 
Total • 



Norn I na I 
Agreement 



Marginal 
Agreement 



Llqht«s 
Extens I on 
of Kappa 



Individual 




1 


♦ 


1 


0 


0 


0 


0 


Pa i r 




2 




0 


0 


0 


0 • 


' — 0' 


Smel I OrouP 


(3-8) 


3 




0 


0 


0 




y 


large Group 


(>8) 


k 


* 


22 


0 


0 


28 


i 


None of the 


above 


5 




0 


0 


0 


0 


2 



MarQinel Totals ♦ 



23 



28 



♦ 



i 

o 

0 
51 
2 

54 



♦ 

♦ 

♦ 
♦ 

♦ 
♦ 



l«000 



.549 
1 #000 



• 043 



LOCO 



■ 549 
♦ 667 



t 063 
i t 000 



7Ji 



7;jo 



category Marginal Agreement! .420 
Cohen's Kappal ,149 



4 




Table 391 

Phase *V - IQE Evaluation Project 
DascriPtlve study - Spring 1978 



Re If ab I M t y on 
Observer 30 
School 433 
Week 4 



Mater I a 1 1 
Expert 30 
Grade 2 
Day 4 



Material! PaPer/pencll 



Catego-y chosen 
by Expert 80 



Cat. 
* 



Category csosen 
by Observer 30 



Marg I na I 
Total s 



Now I na I 
Agreement 



Marg I na | 
Agreement 



L | ght i s 
Extent | on 
of Kappa 



used 

Not used 
Other 



1 



♦ 

2 ♦ 
♦ 

3 ♦ 
♦ 



Marg | na I Tote I s ♦ 

♦ 



51 
0 
0 

51 



0 
0 
0 



I 
0 
2 



♦ 
♦ 
♦ 

♦ 

* 

♦ 
♦ 



52 
u 
2 

54 



♦ 
♦ 
♦ 

♦ 
♦ 
♦ 

♦ 
♦ 



,981 



1,000 



981 



667 



,654 



1,000 



732 



9 

ERIC 



\ 



/ 



category Marginal Agreement I tFj24 
Cohen's Kappa! 



4> 



733 



Table 392 

Rhe«e IV • JOE Evaluation Project 
Deter iPtlve Study - spring 1978 

Reliability on Learner wovei 
Obtarver 30 Expert *0 
School 433 Grade 2 
week 4 Oay * 
OMR 



4> 



Category choien 
by Expert 80 



Engaged - wr I t ten 
Engaged - orat 
Engaged - covert 
Not engaged » Inter lm 
Not engeged • welting 
Not engaged -off tetfc 
Nona of the above 



Category chosen by Observer 30 



Marginal totals 



Marginal 
Toteie 




Nominal m Marginal 



Uloht»t 
EKteni Ion 



Agreement * Agreement 0* KePPe 



1 882 

• 500 

• 381 
,667 

,tso 

1,000 
it^OO 



,944 

•soo 

,571 
,66T 
• 619 
,200 
,66? 



,82^ 
,491 
,204 
,684 
,6T1 
1,000 
ItPOO 



category Marginal Agreement! #898 



Cohenti Keppaf ,841 



734 



. — Table m — — - 

Phase XV - lot Evaluation Project 
Descriptive Study - spring 1978 

reliability on interaction! 
Observer 30 Expert SO 
School *33 Grade 2 
Week M Day 4 

OMP 



Datal 06-25-T9 
Program! WEIP9S 
Data Files RELD8 



Category chosen 
by Expert so 



Target to Teacher 

Target to Othar Adult I ♦ 

♦ 

Targat to Student 
Targat to Tchr i Orp 
Teachar to Targat 
Cthgr Adult tc Targat 6 ♦ 
Studant tc Target 
Teacher to Group 
Other Adult to orouP 
Student to sroup 
hone o* the abova 



Category chosen by Observer 30 
3 * 5 6 T S 9 



Marginal 
Totals 




U|ght»« 

Nowtnai Maroinai Extension 
Agreement Agreement c# Keppa 



.500 



y t 500 



• 491 



>*17 



• 503 



,330 



• 950 



• •TO 



.553 



1 M ^ 
/ .ll) 



Category Marginal Agreeaonti *55i 
Cohen's Keppei #*»5 



73 (J 



4 ■ 



9 

ERIC 



4> 

00 



Table 394 

ph»»a IV - I0E evaluation Projtci 
Ottcrlptlv© Study - spring 1978 

wa 1 1 ab 1 1 1 1 y on oanarai Con tan t 
Obtorvar 30 Expert 60 
8CK0CJ 033 Grada 5 
wo«* 16 Day 2 
OMP 



Cattgorv choian 
by Expart «0 



Cat, 
* 



Category choaan by Observer 30 



Walt 


1 ♦ 


0 


0 


0 


0 


0 


Iran* i 1 1 on 


» ♦ 


0 


a 


0 


r 


0 


Management 


J ♦ 


0 


0 


0 




0 


tr aak 


A ♦ 


J 


0 


0 


0 


0 


Nortacadamic 


5 ♦ 


3 


0 


0 


0 


0 


Other Academic 


6 4 


0 


0 


0 


0 


0 


Nona o# the Above 


7 ♦ 


0 


0 


0 


I 


0 


Mercs I ni! 


Total! ♦ 


0 


1 


0 


1 


0 



0 
0 
0 
0 
0 

c 

0 



0 

t 

1 

0 
0 

93 



Marginal 
Total e 



♦ 

♦ 

♦ 

♦ 
♦ 
♦ 

♦ 

♦ 

♦ 



o 
* 
i 

0 
0 
0 
91 

96 



Ught«8 

Nominal Marginal Extension 
Agraement Agreement 69 KaPPe 



tSOO 

OgOOO 



• 800 
0-000 
0*000 



,489 

0*000 



• 989 



• tTi 



category Marginal Agreementi #370 
Cohen »• Kappat .*•» 



71)7 



Phase IV - 1 GEEvo | uat I on Project 
Descriptive s\udy - SPrlno 1978 

pe I | ab I I I ty on Pace 
Observer 30 . Expert 8 
School 433 Grade 5 
Week 16 Day 2 
OMP 



Category chosen 
by Expert 80 



Cat • 
ft 



Cateaory chosen 
by Observer 30 



1 



Mar 3 I na 
Total s 



Nom | na I 
Agreement 



MarQl nal 
Agreement 



Light 'S 
Extent l on 
of Kappa 







♦ 












♦ 








Set * 


1 


♦ 


20 


1 


0 


♦ 


21 


4 


,952 


1,000 


,939 






♦ 








4 




4 








Other 


2 


♦ 


0 


69 


1 


4 


TO 


4 


,986 


,972 


,943 






♦ 












4 








Ne 1 ther 


3 


4 


1 


2 


2 


♦ 


5 


4 


,400 


• 600 


,381 






♦ 








♦ 




4 












4 








♦ 




4 








Hard 1 na I 


Total s 


4 


21 


72 


3 


4 


96 


4 












4 








♦ 




4 









4 1 4 4 4 * 



category Marginal Agreemanti ,B57 
Cohen's Kappel ,8Ti 



JL- 
\0 



7:)!) 



r 4 4 1 U 



Table 396 



Phase IV - IGE Evaluation Project 
Descriptive Study • Spring 1978 

Re I I ab I I 1 1 v on Group | no 
Observer 30 Expert 80 
School 433 Grade 5 
Week 16 Day 2 
DMP 



o 



Category chosen 
by ExP*rt 80 



Individual 
Pa I r 

S ma I I GrouP t3-8) 
large orouP (>8) 
None of the above 



Category chosen by Observer 30 
Cat, Margina 
u 1 2 3 4 5 Totals 



Norn I na I 
Agreement 



Marginal 
Agreement 



Light's 
Extens I on 
of Kappa 














19 


♦ 


1.000 


,905 


1,000 




4 








2 




.500 


,500 


,495 












2 




.500 


1,000 


,489 




♦ 








68 




1.000 


,986 


1,000 




♦ 








5 




.400 


,600 


,381 




♦ 









96 ♦ 
+++ ++++++ 



Category Marglna 



7 in 



Cohjan*s Kappa I 

/ 



Agreementl ,798 
,883 



7 it 



er|c 



Table 397 



J-naie TV - X6E Evaluation Project 
Datcr lot I v« Study - SprlnQ 1978 

Re liability on Mater I a I s 

Observer 30 Expert 80 

School 433 Grade 5 

Week 16 Day 2 

DMP 



Material l Paper/pencil ) 



Category cbosen 
by Expert 80 



Used 

Not uced 
Other 



Category chosen 
by Observer 30 
Cat. Marginal 
# 12 3 Totals 



1 

2 



♦ 



AO 
0 

1 



1 



49 



1 
2 



♦ 

♦ 
♦ 

♦ 



41 
50 
5 



♦ 
♦ 



Nom| na I 
Agreement 



• 976 

• 980 



400 



Marginal 
Agreement 



1,000 

• 962 

• 600 



Light's 
E*tens | on 
of Kappa 



,957 
,956 
.381 



Marg I na 



Totals ♦ 
♦ 



41 



52 



♦ ♦ 

♦ 96 ♦ 



742 



C«togory Marginal Agreementl #854 
Cohen 1 * Kappai # 902 



H 



743 



Table 398 



Phase IV • IGE Evaluation Project 
Descriptive Study - SPrlno 1978 g 

to 

Re I I ab I I 1 1 y on Mater lets 
Observer 30 Expert 80 
School **33 Grade 5 
Week 16 Day 2 
DMP 



Mater I a I I Game 



Category chosen 
by Expert 80 



Used 

Not used 
Other 



Cat • 
n 



l 
2 
3 



Marginal Totals 



Cateoory cKosen 
by Observer 30 



i 



♦ 
♦ 
♦ 
♦ 
♦ 
♦ 

♦ 



2 


0 


0 


0 


ee 


1 


1 


2 


2 


3 


90 


3 



Mar 0 I na 
Tota I s 



♦ 

♦ 
+ 



2 
89 
5 

96 



♦ 
♦ 

♦ 



Nom I no I 
Agreement 



1*000 
• 989 
#400 



Marginal 
Agreement 



LlgHtts 
Extens ) on 
of Kappa 



•667 1*000 
•989 *820 
•600 .381 



74 



category Marginal Agreementi »752 



Cohen's <appaJ ,676 



Table 399 

Phase IV * IOE Evaluation Project 
Descriptive Study - SPMno 1978 

Rel lab) I I tv ori Material! 
Observer 30 Expert 80 
School 433 Grade 5 
Week 16 Day 2 
OMP 



Material! Audio/visual aid 



Category chosen 
by Expert 80 



used 

Not used 
Other 



Cat, 



Marginal Total? 



Category chosen 
by Observer 30 



93 



MarO I na 
Total s 



Nom I na I 
Agreement 



Mar o I na | 
Agreement 













♦ 








1 




0 


1 


0 


♦ 


1 




0 .000 




















2 




0 


69 


1 




90 


♦ 


• 989 




















3 




0 


3 


H 




5 




• 400 





















96 



♦ 
♦ 



L I ght ' s 
Extens | on 
of Kappa 



0,000 0.000 
,968 i644 
,600 .381 



ERIC 



7415 



category Marginal AQreementl #523 
Cohen's Kappa; .422 



747 



Table 400 



Phase IV • I GE Evaluation Proje t 
Descriptive Study - sprlno 197ft 

Re I I ab II ! t v on Mater l a I s 
Observer 30 Expert Bo 
School 433 Grade 5 
Week 16 Day 2 
DMP 



Material I Other 



Category chosen 
by Exoort 80 



Used 

Not u ,ed 
Other 



Cat t 
U 



Cateqory chosen 
by Observer 30 



1 




M a r o I n a 
Total s 



Marginal 



♦ ♦ ♦ 

1 + C 0 0 * 0 ♦ 

♦ ♦ ♦ 

2 ♦ 1 89 1 * 91 * 
+ + ♦ 

3 ♦ 0 3 2 + 5 ♦ 

♦ + ♦ 
+ + + + + + + + + + + + + 4. + + + + + + + + + + + 4 * + 

♦ + ♦ 
'OtBll ♦ 1 9? 3 * 96 ♦ 

¥ 4 4- 



Nom | ne I 
agreement 



,978 

• 400 



Margi a l 
Agreen- ^nt 



0,000 
,989 
,600 



L I gbt i S 
Extent | on 
of Kappa 



,473 
,381 



( 



Category Marginal Qcirepm«nt « 
Cohen's K a p ^ a « ,421 



• b3 0 



\ 

I 



f 



Table 401 

PhaM IV - IOC evaluation ^ojtct 
DatcMPtlva Study - spring 19TB 

ftaliabllltv on Laarnar Movai 
Obtarvar 30 ExPtrt *0 
School *33 Orada 5 
waa* 16 Day * 
DMP 



Catagcry chotan bv Obtarvar 30 



Catagery e*o«an 
bv EKPa^t 80 



Cngagad - *r I ttan 

Engagad *■ oral 

Engagad - covart 

hot angagad - Intarin 
hot angaoad - waiting 
Not angagad *o#f tatk 
Nona o# tha abova 




1 

2 

3 

4 

5 

6 

7 



Marginal Total* 



11 
0 

0 
0 

\ 

0 



0 

1 
1 
1 

0 

I 

0 



I 

0 
IT 
1 
4 

I 
0 



0 
0 
0 
3 

0 
0 



0 
0 

t 

0 

14 

1 

\ 
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Table 402 

Phete IV - IGF fvaluatlon project 
Descriptive Study • SPMnft 1978 
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Phot* IV - TGE Evaluation. Projact 
Oatcr | n t I vty Study - Spring 1*70 
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Table 404 

Phase IV • IGE Evaluation Project 
Descriptive Study - Spring 1970 

Re I j ob I I i t y on Pace 
Observer 40 Expert 80 
School 440 Grade 5 
week 4 Day 4 
DMP 
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Table 405 

Phase IV • 2GE Evaluation Project 
Descriptive Study • Spring 1978 
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Table h Ob 

Phase IV - K*E Evaluation Project 
DetcrlPtlv* Study - spring 1978 
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Table 407' 

Phase IV - I6E Evaluation Project 
Descriptive Study - Spring 1978 

Re I lab r I I ty on Mater lal s 
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Phase IV - IftE Evaluation Project 
) Descriptive Study - Spring 1978 
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Phaie IV • IOE EveTuatlon Project 
Descriptive studV • Spring 1976 
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Table 410 

Phase Iv - IOE Evaluet ion Project 
Descriptive Study spring 1970 

Reliability on interactions 
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